At my current role, I’ve had the opportunity to witness and contribute to several significant technology shifts. When I joined my company back in late 2018, a robust ecosystem of internal Python libraries had already been established, primarily built around Flask, aimed at accelerating application development by sharing common functionalities across projects.
However, over time, we’ve encountered persistent friction. Seemingly minor updates that should be straightforward became monumental tasks, largely because our internal libraries were tightly coupled and riddled with interdependencies. As our tech stack evolved and our workforce shifted, many of these libraries lost their original maintainers and fell out of active support, leaving us with a difficult maintenance burden.
Interestingly - and unfortunately - as we gradually adopted Go, many of these problematic patterns persisted, perpetuating the same tightly coupled and highly interdependent structure we hoped to avoid.
The heart of the issue lies in how we structured our internal libraries. Initially, the idea was appealing: wrap external frameworks and libraries within internal packages to streamline development. But this wrapping introduced several subtle but severe complications. Instead of offering standalone functions or middleware compatible with standard frameworks, these “wrapped” packages often became opaque and rigid, making simple updates incredibly complex and creating hidden dependencies.
Another overlooked consideration was our transient dependencies - libraries our code indirectly depended upon. Without careful management, we found ourselves inadvertently depending on outdated or insecure versions, complicating upgrades and patching efforts. Tools like Dependabot, which automatically bump transient dependencies, sometimes obscured deeper issues by masking what our applications directly depended on.
Recognizing these pitfalls, we’ve now established several guidelines to help us move forward and prevent repeating past mistakes:
-
Avoid Wrapping External Libraries:
Instead of encapsulating third-party frameworks (e.g., FastAPI, Flask) within internal packages, we aim to provide libraries of reusable functions or middleware that can seamlessly integrate with standard frameworks. This approach provides flexibility and significantly simplifies dependency management. -
Leverage Standard Libraries Where Possible:
Rather than defaulting to third-party solutions, we now encourage assessing whether language standard libraries can handle common tasks effectively. For example, Go’s built-in structured logging (slog) is perfectly adequate for most use cases, reducing unnecessary dependencies. -
Prioritize Well - Maintained Third - Party Libraries:
When external libraries are required, we actively choose mature, community supported options. Tools like Snyk Advisor assist us in evaluating the long term health and security of these dependencies. -
Explicitly Manage Dependencies:
Understanding and explicitly managing our application’s direct and transient dependencies prevents entanglements. Tools like Deptry help ensure that dependencies are clear, manageable, and directly reflect our application’s requirements.
Our current strategy involves actively “ejecting” from tightly coupled internal frameworks, pulling critical functionality directly into our repositories where possible, and incrementally refactoring to eliminate unnecessary dependencies. By progressively decoupling libraries and clearly defining dependency boundaries, we hope to significantly reduce technical debt.
Ultimately, these experiences have taught us an essential lesson: decisions made today resonate into the future, long after their original architects have moved on. A more modular, explicit, and disciplined approach to dependency management and library usage doesn’t just reduce maintenance costs - it empowers us to build sustainable, robust systems.
Like the Scouts say, strive to leave your tech ecosystem better than you found it.
