In software development and other information technology fields, technical debt (also known as design debt or code debt) is the implied cost of future reworking because a solution prioritizes expedience over long-term design. Analogous with monetary debt, if technical debt is not repaid, it can accumulate "interest", making it harder to implement changes.
Convincing a non-technical manager to reserve some portion of a sprint for 100% technical tasks, where the short-term gains aren't clear, is a really hard task for a developer. The developer completely understands what CI/CD is and knows it will speed up project delivery. Non-technical managers, however, are focused on delivering the next feature. They have promised to deliver Feature A by Date X, and their job is to ensure that it gets delivered. Part of their role is to keep developers focused and make sure they don’t stray down the "tech fascination path."
This is a point of frustration for both sides, and it's hard to establish clear communication. Developers see the technical improvement as an obvious win and get frustrated when the manager doesn’t understand this. On the other hand, managers, since they don’t understand the technical details, get frustrated that the developers aren’t focused on delivering the promised features and instead seem more interested in doing "cool" stuff.
My advice to developers is to try explaining these technical improvements to your managers using numbers. Keep track of the time you spend on manual tasks that could be automated, and take note of all the errors that happen but could have been avoided if technical debt were reduced. At DareData, we were called in to solve a project where a data engineer was spending 100% of their time running a machine learning model on a daily basis.
My advice to managers is that controlling technical debt is key to maintaining efficiency in project deliveries. Sure, you can take shortcuts and deliver a feature without that fancy tech piece, but if you consistently rely on shortcuts, you will lose in the long run. Push your developers to explain their proposals in simple terms, and don’t assume right away that they’re just trying to play with the latest cool tech.
In data engineering projects, technical debt often costs more time on a daily basis than in traditional software engineering projects. The code in data engineering projects takes much longer to run. Because of accumulated technical debt, there are data engineers whose daily job is to wait for one script to finish running so they can launch the next one.
If you want help to reduce the technical debt on your data engineering pipelines reach me out at rui@daredata.ai