Can Unit Tests be black box tests?

I've been thinking a while about unit tests, which are supposed to be the base of the testing pyramid – in other words, the majority of tests should be unit tests.

My first thought is about black-box-testing. If our unit tests relies on internal details of the code, it will be much more brittle and often not that useful when refactoring, because they break and I will need a completely new test, which adds another possibility of error (because I could miss something important that was tested before but now isn't any more).

Of course, if my unit test has no external dependencies (outside the unit), then I can write black box test by simply calling the public api of my unit and verifying the result. Easy, peasy. But often, we don't have that. Often, our units call other units and then we need stubs, mocks, test doubles, etc.etc. to include them into our test. And by doing so, we add "knowledge" of the internal working of the code into our test. And when this internal working changes – even if the outside contract still works – the test breaks.

Thus, unit tests for units with external dependencies are per definition brittle, not robust to purely internal changes. Unfortunately, it also doesn't seem possible to write a unit test in a way that is agnostic to the usage of external dependencies.

Of course, the other extreme is an integration test, which wouldn't care about the internal workings at all – but takes a lot longer and thus isn't a good replacement for the bulk of unit tests. So an integration test is more robust to change and much more suited to test the public api – but only that of a much bigger unit, which makes it much slower.

Is there any good way of solving that dilemma? Can we write unit tests to be robust to purely internal changes? Or do we have to live with the fact that unit tests have to change when we refactor internal stuff? But how to prevent regressions then? On a much bigger level, integration tests can prevent regressions, but they are unsuited for all the little details, edge cases, etc. due to their lack of speed.