Yes. US immigration law is primarily concerned with where you are physically sitting while you do the work, not where the company is registered.
We frequently hear from founders in F1, L1, or H1B status who are eager to begin working on a new company, but unable, for a variety of reasons, to have that company sponsor them for a work visa. They often suggest this workaround: register the company in another country to get things going until they have the resources to arrange for a US work visa. The theory is that if the company and its bank account are outside of the US and using a foreign currency to operate, then US immigration law shouldn't be relevant to work for this company, even if that work is done from within the US.
This is not the view that US immigration law takes. If you are physically in the US, then you need US authorization to work, even for an employer that is outside of the US. Without work authorization for your startup, your activities while you are physically in the US should be limited to the "business" activities that would be allowed on a B1 business visa. As soon as you physically leave the US, however, these restrictions no longer apply.