DEV Community: Middleware

The 2024 DORA Report: State of DevOps Breakdown Summary

Shivam Chhuneja — Wed, 06 Nov 2024 12:49:05 +0000

For the past 10 years, we have seen Accelerate State of DevOps: Report released annually, built upon the insights and data from thousands of industry respondents. The 2024 DORA Report was published recently.

As you can probably guess, the report is packed with data and insights that not only give a deep dive into software delivery and operations but also pack a lot of value for Engineering Managers, Engineering Leaders and Software Engineers.

So, we thought why not take our notes from this year's DORA Report, expand on them a little bit and share them with you in an article. This blog provides a summarized view of the report's findings, highlighting what matters most for team productivity, AI integration, and platform engineering.

Of course since this is a summary it cannot cover all the nuances that the DORA team have covered in their complete report, so while we do think this article is extremely valuable we still believe that just going through this blog will not be enough especially when it comes to context and methodology.

"Granny sighed. "You have learned something," she said, and thought it safe to insert a touch of sternness into her voice. "They say a little knowledge is a dangerous thing, but it is not one half so bad as a lot of ignorance."\
-- Terry Pratchett, Equal Rites

If you want exceptional visibility & actionable insights for your org's software delivery process, Middleware is your go to tool to get started within minutes & start a process transformation!

1. State of Software Delivery Performance

Performance benchmarks this year look as crazy as they do every year if we compare elite teams to low performing teams.

Chart from The 2024 DORA Report, Pg 14

Elite teams show unparalleled efficiency and recovery rates, which definitely say something about the value of mature DevOps practices.

127x faster lead time for changes
182x more frequent deployments
8x lower change failure rate
2293x faster failure recovery time

Elite teams set a high bar, showing how consistent, high-velocity performance can dramatically improve software delivery outcomes.

However, we loved what the report says about elite performance: "The best teams are those that achieve elite improvement, not necessarily elite performance"

This is something we have mentioned in our articles around DORA Metrics as well. 2 distinct teams, of distinct size, delivering distinct software, to distinct users should not be compared in absolute terms without context.

2. AI's Role in Software Delivery

AI adoption is picking up rapidly, yet concerns around trust in AI-generated code remain.

Chart from The 2024 DORA Report, Pg 20

This year's report dove into how teams are adapting to AI integration and its impact on productivity.

39% of developers report little or no trust in AI-generated code quality
81% say their organizations have shifted priorities to increase AI incorporation into applications
At an individual level 75.9% are relying on AI in things like writing code, summarizing info, documentation, writing tests etc.

Building trust and transparency in AI tools is essential as organizations increasingly adopt AI in development workflows. Even though AI is helping people do meaningful work with increased productivity the overall sentiment still remains somewhat of concern.

One participant even likened the need to evaluate and modify the outputs of AI-generated code to "the early days of StackOverflow, [when] you always thought people on StackOverflow are really experienced, you know, that they will know exactly what to do. And then, you just copy and paste the stuff, and things explode " (P2). --- 2024 DORA Report, Pg 24

3. Platform Engineering's Impact on Developer Experience

Platform engineering transforms developer workflows, enabling self-service and reducing friction.

Platforms behave in a similar way as transformation efforts, the early effects tend to be positive with a dip in the mid-term and recovery as the internal platform matures.

Chart from The 2024 DORA Report, Pg 50

8% increase in individual productivity with internal platforms
6% gain in productivity at the team level
And here we go: an 8% decrease in throughput & 14% decrease in change stability!

While platforms smoothen out the processes, they introduce new layers of complexity that can impact throughput and stability. Increased handoffs and dependencies may hinder speed.

Platform teams should balance automation with flexibility to prevent these operational slowdowns.

Overall, internal platforms show great promise in boosting productivity and team efficiency across development organizations however they are not a cure-all!

4. Developer Independence and Self-Service Workflows

Developer autonomy, a core principle of platform engineering, correlates strongly with productivity gains.

Self-service capabilities reduce dependencies on enabling teams and accelerate project timelines(owing to less handoffs and touchpoints within the process).

5% productivity gain at individual and team levels for developers without an "enabling team"

Interestingly, the impact on productivity of having a dedicated platform team was negligible for individuals. However, it resulted in a 6% gain in productivity at the team level. This finding is surprising because of its uneven impact, suggesting that having a dedicated platform team is useful to individuals, but the dedicated platform team is more impactful for teams overall.

Since teams have multiple developers with different responsibilities and skills, they naturally have a more diverse set of tasks when compared to an individual engineer. It is possible that having a dedicated platform engineering team allows the platform to be more supportive of the diversity in tasks represented by a team. --- 2024 DORA Report, Pg 52

5. The Role of Documentation in Developer Productivity

While Agile emphasizes "working software over documentation," DORA's findings highlight that quality documentation is essential for effective development.

Strong documentation isn't just about quantity but ensuring content is findable and reliable. A user-centered approach to documentation supports developer independence and enables smoother workflows.

Chart from The 2024 DORA Report, Pg 63

Focus on findability and reliability to keep documentation useful
Promote a sustainable documentation culture that maintains relevance
User-focused documentation amplifies technical capabilities and organizational impact

In short, well-maintained, user-centered documentation is foundational to productivity.

6. User-Centric Focus and Transformational Leadership

Focusing on the user in software development yields notable gains, with transformational leadership playing a significant role.

Leaders who empower teams and align projects with user needs increase productivity, satisfaction, and overall organizational performance.

User-centered development correlates with a 40% boost in organizational performance(2023 metric)
Strong transformational leadership(increasing transformational leadership by 25%) leads to a 9% productivity increase

Organizations combining user-centric practices with transformational leadership push team towards success and impactful, user-aligned products.

Let's Wrap This

The 2024 DORA Report highlights that robust DevOps practices, responsible AI integration, and thoughtful platform engineering are key to high performance.

For teams looking to achieve these benchmarks, Middleware offers a straightforward, quick way to measure and refine your DORA metrics, Project Flow Metrics, Bottleneck analysis out of the box to refine your software delivery practices.

Electron Repo—Unpredictable Metrics, Efficient Deployments

Rajni Rethesh — Mon, 30 Sep 2024 11:18:29 +0000

There's a high probability that a bunch of desktop apps you are using are built with Electron. Electron is a framework that combines Node.js, parts of Chromium, and a layer of native code. Apps like Visual Studio Code, Slack, Atom, WhatsApp, or even the installer for Microsoft's Visual Studio use it to build for Windows, macOS, and Linux. Though Electron has its own fanbase, many flock to its competitor, Tauri, for its significantly better startup times and lower memory consumption.

Having said that, with over 114k stars and 15.3k forks, and a thriving community of 1300 contributors, Electron is not going anywhere anytime soon.

As part of our 100 days 100 case studies of top Github repositories, we also toyed around the Electron repo, analyzing their dora metrics using Middleware Open-source.\
Check out how Middleware helps track a project's engineering pipeline in this live demo.

If you're excited to explore these insights further and connect with fellow engineering leaders, come join us in The Middle Out Community and subscribe to the newsletter for exclusive case studies and more!

Understanding Electron Dora Metrics: The Domino Effect

Team Electron displayed a good performance in the months of June and August with a great deployment frequency of 92 and 170 releases. This shows that the team follows an aggressive continuous deployment strategy. However, July showed a sluggish movement with only 6 deployments.

Digging a little deeper, it was found that the biggest bottlenecks in their development cycle in July were the first response time, rework time, and merge time. All three were way above the benchmark set by the 2023 State of DevOps Report. None of these even came close to meeting the standard, making these three areas the major blockers holding things up.

Let's see the numbers:

First Response Time: Back in June 2024, it took about 25.12 hours to respond. Then, it shot up to 68.17 hours in July, only to drop back down to 20.02 hours in August.

Rework Time: June's rework time was 20.23 hours, but things went wild in July, skyrocketing to a whopping 90.79 hours, before finally cooling off to 14.38 hours in August 2024.

Merge Time: In June, merge time was 35.26 hours, then it blew up to a crazy 282.73 hours, only to fall to a much better 16.53 hours in August.

And just like a domino effect, the wild fluctuations in these three metrics really threw off their cycle and lead time.

Also read: Hugging Face and DORA Metrics: Fast Code, Slow Response

Electron Engineering Pipeline: Busy Putting out fires

Delays in code reviews, along with the back-and-forth nature of iterations and corrections within pull requests (PRs), can significantly contribute to the extended first response and rework times. When code reviews lag, it creates bottlenecks, forcing developers to spend more time on revisions rather than moving forward with new tasks.

Another significant factor is the disproportionate amount of time allocated to bug fixes---around 45% of the total development cycle. This high percentage suggests that developers are spending nearly half their time putting out fires instead of focusing on innovation. As a result, critical resources that could be directed toward launching new features and enhancing performance are instead diverted to addressing bugs.

In fact, only 25% of their time is spent on performance improvements, and a mere 20% goes towards developing new features. This imbalance highlights a reactive rather than proactive approach to development, ultimately stalling progress and preventing teams from achieving their strategic goals.

Also read: Node.js Dora Metrics: Faster Deployments, Slower Merge Times

The Key Strengths They Can Cash In On

Robust CI/CD Pipelines: The repo's loaded with workflow files like build.yml, pipeline-electron-build-and-test.yml, and pipeline-segment-electron-gn-check.yml. These files basically handle testing, building, and deploying automatically, cutting down on the time developers would normally spend doing things manually. With less manual intervention, teams can push out changes faster, which helps reduce those long first response and rework times. By speeding up these processes, merge times will naturally shrink too.

Frequent Smaller Changes: Instead of dropping massive, monolithic updates that can lead to a ton of back-and-forth, the team is rolling out smaller, bite-sized changes more often. This means PRs are more focused and easier to review, like PRs #43207 and #43332 .

The result?

Fewer issues to fix, faster first responses, and quicker rework cycles. Smaller, frequent changes lead to smoother merges since you're not dealing with a mountain of code all at once.

Comprehensive Automated Testing: The workflows also have a heavy focus on automated testing, with files like pipeline-segment-electron-test.yml and test.yml keeping code quality in check.

Automated testing ensures that new code doesn't break anything major, which means fewer bugs, less rework, and shorter review times. With solid testing in place, they can catch errors early on, which means smoother sailing when it comes to merging PRs later down the line.

Also read: What Developer Productivity Metrics Should An Engineering Manager Really Track?

Leaving Thoughts: Electron Repo---Unpredictable Metrics, Efficient Deployments

Electron should really lean into their active, thriving community and strong CI/CD pipeline to push their product through that final stretch. They also need to step up their documentation game---it'll make life a lot easier for developers, helping them tackle projects with more confidence and efficiency.

If you find these learnings interesting, we'd really encourage you to give a shot at Dora metrics using Middleware Open Source. You could follow this guide to analyze your team or write to our team at productivity@middlewarehq.com with your questions and we'll be happy to generate a suggestion study for your repo --- free!

Also, If you're excited to explore these insights further and connect with fellow engineering leaders, come join us in The Middle Out Community and subscribe to the newsletter for exclusive case studies and more!

Did you know?

Electron was originally created by GitHub for their desktop app, Atom. Since then, it's grown into a framework used by huge names like Microsoft, Discord, and even Facebook!

Further Resources

TypeScript Dora Metrics Analysis: They Just Need a Smart Game Plan!

Rajni Rethesh — Fri, 27 Sep 2024 04:33:16 +0000

Ask any developer, and they'll tell you: catching errors during the development phase is a game-changer in the development cycle.

As an answer to their prayers, Typescript came into existence. Developed by Microsoft, TypeScript is a high-level, open-source programming language that allows developers to identify and fix errors early, rather than waiting for runtime to strike.

This proactive approach significantly cuts down on bugs and enhances overall code quality. With its emphasis on safety and maintainability, TypeScript has become a core option for projects of all sizes.

Plus, with a thriving community of nearly 800 contributors continuously improving the language, TypeScript is almost a movement! To gain deeper insights into this movement, we analyzed their engineering Dora Metrics using Middleware Open Source. Keen to know what we found?

If you're excited to explore these insights further and connect with fellow engineering leaders, come join us in The Middle Out Community and subscribe to the newsletter for exclusive case studies and more!

1. Deployment Frequency, Merge Time, First Response Time & Rework Time Analytics

TypeScript powers through rapid iterations and delivery, but the surge in merge time---from 1.76 days in June 2024 to 3.39 days in August 2024---indicates a major slowdown in review times, likely caused by complex pull requests or a backlog in reviews.

Let's delve deep and see what story their number narrates:

June 2024: They had a good 92 deployments

July and August 2024: They hit a deployment slump in July with 26 deployments but managed to claw their way back up in August with 66 deployments.

It's like they pressed pause, then fast-forwarded to resume their usual deployment groove.

Upon closer examination, we observe a steady decrease in the cycle time, dropping from 12.56 days in June to 11.09 days in August 2024, and a reduction in lead time from 13.0 days in June to 11.42 days in August.

However, the fluctuations in first response time, merge time, and rework time raise some concerns that need to be addressed.

Merge Time: In June, it was 1.76 days, which reduced to 0.99 days in July and then increased to 3.39 days in August.
First Response Time: It started at 4.11 days, climbed to 8.05 days, and ultimately decreased to 3.01 days.
Rework Time: Initially, it was 6.59 days, then it dropped to 0.45 days, before rising again to 4.68 days.

The instability in these metrics signals potential roadblocks in their workflow. The rising merge time suggests that while deployments may have increased, the review process is becoming a bottleneck. Meanwhile, the erratic first response and rework times indicate that the team may be struggling to keep pace with review demands and address issues promptly.

Also Read: How Bootstrap maintained a robust merge time

2. Their Strengths: A Powerful Ecosystem and A Solid Backing

Microsoft Backing: TypeScript is maintained by Microsoft, ensuring robust support and regular updates, which adds to its credibility and reliability.

Active Contributions: With over 749 contributors, the repository benefits from a diverse and active group of developers who continuously improve and maintain the project.

Extensive Ecosystem: A vast ecosystem of libraries and tools can help automate and streamline various aspects of the development and deployment process, leading to more frequent deployments. Also, the availability of a wide range of tools can help developers implement changes more efficiently, reducing the lead time for changes.

Robust CI/CD Pipelines: Automated testing and deployment ensure that changes can be reliably deployed frequently without manual interventions.

Clear Contribution Guidelines: The documentation and guidelines, present in files like CONTRIBUTING.md, set clear expectations and reduce friction for new contributions.

Also, check out where Hugging Face shines the most

3. Leveraging Strengths to Build a Robust Engineering Roadmap

Capitalize on Active Contributions

With over 749 active contributors, the wealth of ideas and improvements should not be underestimated. Actively engage the community and foster collaboration, enabling the team to rapidly iterate and enhance features, creating a continuous cycle of improvement.

Bug Fixes: Taming the Beast with Quality Code and Smart Strategies

The TypeScript repo currently spends about 35% of its time fixing bugs. To bring this number down, they should leverage their automated testing expertise to catch issues before they escalate. By implementing unit tests, integration tests, and end-to-end tests, they can ensure that new code doesn't introduce bugs and that all aspects of the application are thoroughly covered.

Strategies to Resolve Delays in Feature Pushes

They spend 40% of their time and efforts in feature pushes and to ensure these features are incorporated and finally deployed into production without delay and hiccups, they can follow the below-given steps:

1. Prioritize Communication

Implement short daily stand-ups to keep everyone updated on tasks and blockers.

2. Streamline the Review Process

Establish and iterate clear code review guidelines to speed up the process.
Assign reviewers to pull requests (PRs) right after they are created to prevent bottlenecks.

3. Automate Testing and Integration

Set up a strong CI/CD pipeline with automated tests for each PR to catch issues early.
Ensure high test coverage for new features to validate functionality before merging.

4. Set Realistic Deadlines

Break down complex features into smaller tasks for easier tracking and identification of delays.

5. Conduct Retrospectives

Hold post-mortem discussions after each feature rollout to identify successes and areas for improvement.

Our Take: TypeScript Has the Tools to Build a Winning Roadmap!

Team TypeScript has all the necessary ammo it requires to build a powerful engineering pipeline. It just needs to leverage it strategically to foster a positive cycle of productivity.

With continued focus on refining their processes, the TypeScript team is well-equipped to enhance their performance even further and set a benchmark for others in the community.

If you find these learnings interesting, we'd encourage you to give a shot at Dora Metrics using Middleware Open Source. You could follow this guide to analyze your team or write to our team at productivity@middlewarehq.com with your questions and we'll be happy to generate a suggestion study for your repo --- free!

We at Middleware really enjoy sharing our insights about productivity from the best teams and would feel really encouraged if you could share with your community/friends as well.

Before you go, here's TypeScript Trivia!

Microsoft provides an online tool called the TypeScript Playground, where developers can write and test TypeScript code directly in their browser. This interactive environment allows users to see how TypeScript compiles to JavaScript in real-time.

Flutter’s Engineering Pipeline: A Great Workflow with Minor Hiccups

Rajni Rethesh — Thu, 26 Sep 2024 04:39:15 +0000

Flutter gained a stronghold in the market with its ability to build beautiful, natively compiled applications for mobile, web, desktop, and embedded devices from a single codebase. Many top companies such as BMW, Alibaba, and Google use Flutter for their flagship apps. And it's not just the big players; developers love it too, with over 1,416 contributors and nearly 165k stars on GitHub.

If you're excited to explore these insights further and connect with fellow engineering leaders, join us in The Middle Out Community and subscribe to the newsletter for exclusive case studies and more!

How did this framework, launched by Google in 2017, climb the ranks so fast? Check out the Annual StackOverflow Survey Report 2023

What best practices have fueled its rise?

Where might this open-source repo fall short?

With Middleware OSS, we set out to analyze Flutter's engineering Dora Metrics.

Flutter's Engineering Pipeline: Robust Yet Vulnerable

Flutter's engineering pipeline seems robust with its high deployment frequency. Comparing it to the benchmarks established in the 2023 Accelerate State of DevOps Report, Team Flutter's performance stands out as exceptional.

However, July brought a significant anomaly, marked by a steep decline in Deployment Frequency, plummeting from 253 deployments in June to just 85 in July, before rebounding to 159 in August. Check out some effective strategies to improve deployment frequency

Several potential reasons for this anomaly could be:

1. High Rework Time:\
The significant jump in rework time, from 18.99 hours in June to 50.34 hours in July, suggests that more code had to be sent back for revisions or fixes. This could be due to:

A sudden influx of lower-quality code, requiring more fixes.
Inconsistent or delayed code reviews, causing defects to slip through and then be caught later in the cycle.

2. Longer Lead and Merge Times:\
Lead time surged to 133.57 hours, and merge time extended to 44.79 hours in July.

This points to a bottleneck in the approval and integration process, either due to:

Resource constraints (e.g., fewer developers or reviewers available during this period).
Review complexity, where the code changes were more substantial or complicated, requiring extra time for analysis and approval.

3. Community Engagement Fluctuations:

Open-source projects often see variations in contributor activity due to external factors like contributor availability, interest levels, or major events.

In July, the Flutter repo may have experienced a drop in active contributors or maintainers, leading to slower review cycles and fewer deployments.

Flutter's Strengths and Best Practices

Flutter has quickly established itself as a powerhouse in the app development ecosystem. A few key factors contribute to its strong performance and rapid growth:

Code Reusability: Write Once, Use Everywhere

Flutter's capability to share a single codebase across multiple platforms (iOS, Android, web, etc.) significantly reduces redundancy in development. By writing code once and deploying it across various environments, developers save valuable time and effort, leading to quicker feedback cycles and a consistent user experience.

CI/CD Made Easy: Smooth Sailing for Releases

The seamless integration of Flutter with continuous integration and continuous deployment (CI/CD) pipelines automates the testing and deployment processes. This automation ensures that every code change is tested and deployed efficiently, minimizing manual errors and speeding up release cycles. As a result, teams can respond swiftly to user feedback and release improvements or new features more frequently.

Also read: How to Leverage DORA Metrics to Optimize Your CI/CD Pipeline

Frequent, Low-Risk Changes

Instead of relying on massive feature drops, Flutter's community focuses on small, incremental updates that reduce the risk of bugs and ensure a steady deployment rate. This practice of delivering smaller, manageable changes enables faster feedback and more responsive iteration, which is crucial in open-source environments.

Active Community Engagement

Flutter's vibrant open-source community is one of its greatest assets, fueling rapid evolution through continuous contributions from over 1,400 developers. This active participation ensures quick issue resolution, regular feature proposals, and a fast development pace.

The efficiency of Flutter's review process is evident in pull requests (PRs) like #151126, which improved documentation with a cycle time of just 22 hours and 56 minutes, and PR #151073, addressing system-related fixes in 1 day, 20 hours, and 44 minutes. This swift handling of both minor and technical updates showcases a well-organized review system that supports fast, incremental changes, keeping Flutter both cutting-edge and stable.

Steady Deployment Cadence

With an active community making frequent contributions, Flutter maintained a steady cadence of deployments. Despite a dip in deployment frequency in July 2024, Flutter managed to push changes at a high rate, keeping the project moving forward smoothly. The combination of active contributors, automated CI/CD pipelines, and small, manageable updates enables this pace.

Improving Performance: How Flutter Can Prevent Future Anomalies Like July's Dip?

Flutter's strong engineering pipeline is evident in its regular high deployment frequency, efficient CI/CD integration, and vibrant community. However, the dip in deployment frequency during July points to areas for improvement that can enhance its overall performance.

1. Backup Reviewers During Community Downtimes

The spike in rework time could be attributed to a temporary slowdown in contributions from the Flutter community. While Flutter's community is typically active, the dip in July suggests a need for dedicated expert code reviewers, with backup reviewers ready to step in during slower periods like this to maintain consistency.

Automate Workflows, Distribute Complex Reviews

July saw a noticeable increase in lead and merge times, pointing to bottlenecks in the approval and integration processes. A solution to this problem is introducing automated approval workflows and distributing complex reviews among senior developers. Automating simpler checks will allow human reviewers to focus on more complex code, reducing lead and merge times while keeping code quality intact.

3. Host Hackathons; Recognize Contributors to Boost Community Engagement

Community engagement is one of Flutter's key assets, driving rapid development. However, fluctuations in contributor activity, such as in July, can lead to slower review cycles and fewer deployments. To counter this, Flutter can maintain contributor enthusiasm with regular hackathons, contributor recognition programs, or clear release goals. Keeping contributors motivated and engaged will prevent slow periods in development, leading to steadier deployment rates.

4. Make CI/CD Pipeline Efficient

Flutter's CI/CD pipeline is well-integrated, but the dip in deployment frequency suggests the system may not handle large volumes or complex updates as efficiently as needed. Improving the CI/CD pipeline's capacity by optimizing test suite performance and implementing parallel testing will allow Flutter to manage heavy workloads without slowing down deployments. By scaling up the pipeline, even during busy months, Flutter can maintain its impressive deployment cadence.

Also, check out some other interesting case studies of God-level repos. What are they doing right? What are the hiccups they are facing and learn from the best!

Dora Metrics Rating

On the Dora Metrics scale, Flutter scores a solid 8 out of 10. Their deployment frequency was impressive, showing strong performance despite a dip in July. They shifted gears in August 2024, maintaining lead times that, while fluctuating, stayed within the benchmarks set by the 2023 State of DevOps Report.

However, their primary weakness lies in rework time, which spiked in July. Although it decreased in August, it still remained above the benchmark. By streamlining their rework process, they could significantly improve their overall performance.

Flutter's Engineering Pipeline: A Great Workflow with Minor Hiccups

Flutter's engineering pipeline performs exceptionally well due to its strategic emphasis on collaboration, code reusability, and streamlined processes. The framework's ability to share a single codebase across multiple platforms minimizes redundancy and accelerates development cycles, allowing for quicker feedback and enhanced user experiences.

Moreover, the integration of CI/CD practices automates testing and deployment, ensuring efficient updates and high-quality releases. Coupled with an active and engaged community, these strengths create a robust ecosystem that not only supports rapid iteration but also fosters continuous improvement. This dynamic approach positions Flutter to adapt and thrive in the app development world.

We'd love to hear your thoughts on this case study and its actionable insights! If you're interested in analyzing and monitoring your project's engineering pipeline, give DORA metrics a try with Middleware Open Source.

Feel free to reach out to us at productivity@middlewarehq.com with any questions. We're more than happy to create a custom suggestion study for your repo---absolutely free!

Moreover, If you're excited to explore these insights further and connect with fellow engineering leaders, come join us in The Middle Out Community. and subscribe to the newsletter for exclusive case studies and more!

Trivia

The Flutter community loves to share! There are thousands of packages available on pub.dev, allowing developers to add features and functionalities to their apps with just a few lines of code.

Bootstrap: Strong Merge and Cycle Times, but First Response Time Needs a Revamp

Rajni Rethesh — Tue, 24 Sep 2024 09:54:28 +0000

While Bootstrap is still fondly regarded by many, it's no longer the trendiest tech in town. A few years back, if you asked developers about their go-to front-end framework, chances were they'd rave about Bootstrap. Once the gold standard for styling web pages, it powered a staggering 24% of websites. Its straightforward, powerful design was crafted to speed up development with ready-to-use components and responsive layouts. However, in recent times, it has been somewhat overshadowed by Tailwind, which offers a more utility-first approach that many developers find appealing for its flexibility and customization.

Now that we've established Bootstrap's credentials, let's delve into how its open-source repository manages engineering productivity. Buckle up as we explore a deep dive into the Bootstrap case study---one of the most widely-used front-end frameworks---and uncover the secrets behind its success!

If you're excited to explore these insights further and connect with fellow engineering leaders, come join us in The Middle Out Community and subscribe to the newsletter for exclusive case studies and more!

When it comes to optimizing engineering productivity, Middleware's open-source solution is the benchmark. With detailed metrics on deployment frequency, lead time, MTTR, and change failure rate, it provides unparalleled visibility into workflows.

We took our OSS for a spin by tracking the Dora Metrics of the Bootstrap repository, unearthing valuable insights along the way.

Get ready to explore the key areas of focus in this case study!

Also Read: What are DORA Metrics?

Bootstrap's Key Strengths and Paths for Improvement

Areas Where Bootstrap Shines

Bootstrap has demonstrated remarkable improvements in both Merge Time and Cycle Time.

In June 2024, the merge time stood at a lengthy 159.75 hours (or 6 days and 15 hours). However, by July 2024, this impressive figure plummeted to just 4.77 hours, further decreasing to 3.28 hours in August 2024.

Similarly, while Cycle Time in June was 24.85 days---high but still within the standard Dora metrics benchmark---it saw significant reductions to 5.88 days in July and then 2.68 days in August. Also, if interested, know more about React Native's Cycle Time triumphs .

So, how did they achieve such dramatic improvements?

A combination of quick PR reviews, automated testing, and effective CI/CD integration played crucial roles in accelerating merge times.

For instance, PR #110 showcased swift peer reviews and efficient CI/CD execution, directly contributing to the reduced merge time. Moreover, PR #123 leveraged automated testing to identify issues early, enabling faster merges.

Also read: Check out the exceptional Cycle Time of OpenVino

Area for Improvement: Enhancing First Response Time

When it comes to metrics that demand attention, First Response Time clearly stands out as a critical area for improvement. Analyzing the trends reveals a significant drop from 281.44 hours in June 2024 to 88.22 hours in July, and further down to 61.15 hours in August. Despite these positive strides, the First Response Time remains high, adversely affecting the overall Lead Time.

Key issues include prolonged waits for initial reviews and acknowledgment, as well as fluctuating response times that can create bottlenecks and disrupt workflow consistency.

Our Verdict

After analyzing Bootstrap's performance metrics, we assess its engineering performance at a solid 7. Over the past few months, merge time has seen impressive improvement, plummeting from 159.75 hours in June to just 3.28 hours in August---an excellent sign of efficient code integration.

Lead Time has also made significant strides, dropping from 18.37 days to 2.68 days, reflecting a remarkable enhancement in their workflow. Likewise, Cycle Time has decreased from 24.85 days to 2.68 days, showing that the overall development process has been effectively streamlined.

However, there's still room for growth in First Response Time, which has reduced from 281.44 hours to 61.15 hours. While this is an improvement, it still falls short compared to the benchmark set by top performers highlighted in the Google's Annual Dora Report. Addressing this area could lead to even greater efficiencies and smoother project timelines.
Read: Key Metrics for Measuring Engineering Team Success

Bootstrap's Strength

Its engineering process includes automated testing, continuous integration, and modular design, ensuring high-quality, efficient releases. You can
A highly skilled core team of seasoned web developers oversees the project, manages releases, and reviews pull requests to maintain top-notch code quality.
Bootstrap fosters open collaboration, boasting over 160,000 GitHub stars and tens of thousands of forks, with contributors regularly submitting updates, bug fixes, and enhancements.
Clear contribution guidelines and code review processes ensure that all contributions meet high standards.
Lastly, well-structured project management, transparent roadmaps, and milestones keep development organized. With cutting-edge web technologies, comprehensive documentation, and regular security audits, Bootstrap continues to evolve and remain a highly adopted, reliable front-end framework.

Leveraging Strengths to Overcome Weaknesses

To address the gap in its engineering pipeline, Bootstrap can leverage its strengths in the following ways:

1. Streamlining the PR Review Process

To enhance Bootstrap's efficiency, the team should focus on streamlining the PR review process. Establishing a structured framework with designated reviewers can significantly reduce wait times for initial responses. Here are a few actionable steps:

Define Roles and Responsibilities: Assign specific reviewers to different areas of the codebase. This not only distributes the workload but also ensures that reviewers are familiar with the components they're reviewing. A clear assignment can be based on expertise, allowing for quicker evaluations and decisions.
Implement Regular Review Sessions: Schedule fixed time slots for reviewers to focus solely on PRs. This dedicated time can help maintain momentum and ensure that reviews aren't sidelined by other tasks. For more on optimizing code reviews, see Atlassian's Best Practices for Code Review.
Use a PR Checklist: Incorporate a checklist that contributors can use before submitting their PRs. This can help ensure that submissions are complete and adhere to coding standards, which can reduce the need for extensive back-and-forth during the review process. You can start with a simple checklist like the one found in GitHub's Guide to Pull Requests.

2. Implementing Comprehensive Automated Testing

Automated testing plays a crucial role in catching issues early in the development cycle. This practice not only accelerates the review process but also minimizes back-and-forth communication, leading to quicker merges. Here's how Bootstrap can leverage automated testing:

Integrate Continuous Integration/Continuous Deployment (CI/CD) Tools: Setting up CI/CD pipelines can automate the testing process whenever new code is submitted. Tools like Jenkins, CircleCI, or GitHub Actions can run tests on every pull request, providing immediate feedback to contributors. For more information on CI/CD best practices, check out The Twelve-Factor App and CI/CD for Beginners. Also, learn how to leverage Dora Metrics to optimize your CI/CD pipeline
Create a Comprehensive Test Suite: Develop a suite of automated tests that cover various aspects of the code, including unit tests, integration tests, and end-to-end tests. This ensures that different functionalities are tested and issues are identified early. For guidance on writing effective tests, visit The Ultimate Guide to Automated Testing.
Encourage Test-Driven Development (TDD): Foster a culture of TDD among contributors, where tests are written before the actual code. This approach not only leads to more robust code but also aligns with continuous testing practices. For insights into TDD, see Martin Fowler's Introduction to TDD.

3. Addressing Time Zone Differences

With a global contributor base, addressing time zone differences is essential for maintaining effective communication and collaboration. Here are some strategies Bootstrap can implement:

Coordinate Overlapping Hours: Identify time slots where team members' working hours overlap and schedule important meetings or reviews during these times. This can enhance participation and ensure that crucial discussions happen in real-time. For tips on managing distributed teams across time zones, refer to Harvard Business Review on Managing Global Teams.
Establish "Review Windows": Create specific time frames when contributors are available to review pull requests. Communicating these "review windows" can help manage expectations and improve the flow of feedback.
Utilize Collaboration Tools: Leverage collaboration tools like Slack or Discord to facilitate quick communication among team members across different time zones. These platforms allow for real-time updates and discussions, which can keep everyone aligned. For more on effective communication in remote teams, check out Slack's Remote Work Best Practices.

Conclusion: Bootstrap Excels in Merge and Cycle Time, But First Response Time Remains a Major Weak Spot

Bootstrap has successfully transformed its development efficiency, significantly reducing Merge Time from 159.75 hours to 3.28 hours and Cycle Time from 24.85 days to 2.68 days. These improvements reflect effective collaboration and streamlined processes. However, the First Response Time, currently at 61.15 hours, still poses a challenge, hindering momentum and frustrating contributors. To sustain their progress, Bootstrap should focus on enhancing First Response Time by actively engaging the contributor community, implementing automated acknowledgment systems, and improving communication practices, especially considering the diverse time zones of its contributors.

If you're facing challenges similar to Bootstrap's, consider diving into Dora metrics with Middleware Open Source. Our guide can help you evaluate your team's performance, or you can reach out to us at productivity@middlewarehq.com for a free tailored analysis of your repository.

Also, If you're excited to explore these insights further and connect with fellow engineering leaders, come join us in The Middle Out Community and subscribe to the newsletter for exclusive case studies and more!

Also don't miss out on reading: Top 10 Tips to Get Started with Open Source and GSoC

Trivia

Did you know that Bootstrap now has its own icon library? Bootstrap Icons includes over 1,000 customizable icons, making it easy to add visual elements to your projects without relying on third-party libraries.\

Further Resources

Optimizing OpenStreetMap’s DevOps Efficiency: A Data-Driven DORA Metrics Analysis

Shivam Chhuneja — Mon, 23 Sep 2024 10:47:28 +0000

With the core goal to build a free, editable map of the world, OpenStreetMap's website repo is where all the magic happens.

This open-source project, powered by Ruby, runs on a great community of developers and cartographers across the planet.

We've been looking at some interesting repositories in our 100 days of Dora case studies series and have uncovered some interesting stuff already.

In this case study, we break down OpenStreetMap's DevOps game using DORA metrics, diving into real-world data to uncover how often code ships, how fast changes go live, and what's driving those numbers.

Understanding DORA Metrics

DORA metrics are the go-to for measuring how well software delivery and operations are performing within DevOps teams.

The four key DORA metrics are:

- Deployment Frequency: How often code makes it to production.

- Lead Time for Changes: How long it takes for a commit to land in production.

- Change Failure Rate: The percentage of deployments that break something and need an immediate fix.

- Mean Time to Restore (MTTR): How fast the team recovers from a production failure.

In this case study, we're zeroing in on Deployment Frequency and Lead Time for Changes---two metrics that directly reflect how fast and efficiently an organization delivers.

These numbers give a clear view of engineering speed and process slowdowns, which are crucial to keep improving in today's fast-paced development world.

If you want to dive a bit deeper into what Dora Metrics are and how you can leverage them for your team, you can check out one of our articles.

Key Findings

1. Deployment Frequency

The OpenStreetMap website repository pushed an average of 58 deployments per month over the last three months, signaling a robust culture of continuous delivery and rapid iteration.

Albeit these DF numbers aren't the best, especially considering the other God level repos that we saw, almost 60 average DFs are nothing to be shy about either.

Key Drivers of Deployment Frequency

Robust Automation Pipelines: The repo leans heavily on automation tools like docker.yml, lint.yml, and tests.yml to keep the build, test, and deployment processes running smoothly. With less manual effort and fewer human errors, they've cut down release times significantly.

Efficient Pull Request Handling: June 2024 saw an almost-instantaneous average merge time of just 10.08 seconds! Even with a slight increase in the following months, merge times remained under 6 hours---proof of an agile review process that keeps things moving fast.

Engaged Reviewer Community: Contributors like gravitystorm and kcne are on the ball when it comes to code reviews, keeping the process fast and thorough. Prompt code reviews facilitate quick integration of changes, maintain code quality, and maintain a collaborative development environment.

Pull requests such as #5056 and #5053 are great examples of this active engagement.

2. Lead Time for Changes

While the repository excels in deployment frequency, the Lead Time for Changes---averaging around 13.26 hours over the past three months---shows room for improvement. The first response time is really good as well compared to the averages.

Although a half-day lead time is impressive, especially for an open-source project with global contributors, shortening this could further boost development speed and efficiency.

Key Influencing Factors

First Response Time Variability: The time between a pull request's submission and the first reviewer's response fluctuated significantly.

July saw an average response time of 6.77 hours, compared to just 1.47 hours in June. That's a 4.6x increase. This multiple looks great on an investment portfolio but not much when it comes to the lead time of a repo ;)

These delays in initial feedback can slow down progress and it does compound, adding unnecessary time to the overall process.

Rework Time Fluctuations: Rework time---the period spent revising code after the initial review---dropped from 11.28 hours in June to 2.94 hours in August. It's bittersweet

While the improved rework time shows an improvement in code quality or review efficiency, rework still adds to the overall lead time and offers an opportunity for further optimization.

For example, PR #5016 ("Allow to edit revoked blocks") required significant rework due to its complexity, in turn extending its lead time.

While the repository maintains great lead times, placing more focus on reducing first response times and streamlining the rework process could drive even faster delivery cycles, enhancing both efficiency and development speed.

Diverse Contributions Pushing Growth

The OpenStreetMap website repository thrives on a diverse range of contributions, showcasing a vibrant and healthy open-source ecosystem.

Feature Development (50%): Innovation is at the forefront, with new features driving user engagement and functionality. For instance, PR #5056 added an "og:description" meta tag to diary entries, improving social media sharing and enhancing the user experience.

Bug Fixes (30%): Stability is key, with bug fixes ensuring the platform remains reliable. Notably, PR #5016 resolved a critical issue with user permissions by enabling editing of revoked blocks, improving system integrity.

Documentation (10%): Clear and accessible documentation is vital for the community. PR #5031 updated the "GPX Import email in text format," making it easier for new contributors to onboard and stay informed.

Testing and Quality Assurance (10%): Testing contributions are crucial for maintaining code quality. By focusing on tests, the project ensures that new changes don't introduce regressions, keeping the codebase robust and dependable

Positive Impact on the Project

The deployment frequency and streamlined workflows in the OpenStreetMap repository deliver substantial benefits to both the project and its community of contributors and users.

Accelerated Innovation: With rapid deployment cycles, new features and improvements are rolled out quickly, enhancing platform functionality and user satisfaction. This speed allows the project to stay responsive to evolving user needs and technological shifts.

Enhanced Contributor Experience: Swift integration of contributions motivates open-source developers by validating their work. The efficient review and merge processes foster a positive, collaborative environment that encourages ongoing participation from both new and experienced contributors.

Quality Assurance: Automated testing and continuous integration maintain stability, ensuring that the fast deployment pace doesn't compromise the platform's reliability. Issues are caught early in the process, reinforcing high-quality standards across releases.

Community Trust and Engagement: Regular updates build trust among users, reassuring them of the project's commitment to reliability and progress. This trust strengthens both the user base and contributor engagement, helping the project flourish.

These practices demonstrate how strong DevOps strategies can fuel innovation, improve community involvement, and elevate the overall success of an open-source project.

Strategic Recommendations for Enhanced Performance

To elevate the OpenStreetMap repository's DevOps practices from great to exceptional, here are some targeted actions:

Standardize and Expedite First Response Times

Implement SLA Policies: Set service level agreements (SLAs) for code reviews, such as committing to an initial response within 4 hours.

Automated Alert Systems: Leverage automation to notify reviewers when new pull requests (PRs) are submitted or pending beyond a certain period

Expand Reviewer Pool: Encourage more contributors to join code reviews by offering clear guidelines and training to reduce pressure on a small group.

Reduce Rework Through Enhanced Code Quality

Adopt Pre-Commit Hooks and Checks: Enforce coding standards with tools like linters or static code analyzers before PRs are submitted.

Code Review Guidelines: Create robust guidelines to set clear expectations for contributors, minimizing the need for back-and-forth revisions.

Peer Programming and Mentorship: Promote collaborative development by pairing experienced developers with newcomers, ensuring better initial code submissions.

Foster and Sustain Active Code Review Culture

Recognition Programs: Acknowledge top reviewers with leaderboards, badges, or shout-outs during community meetings to incentivize participation.

Contributor Onboarding: Streamline the process for new contributors to become reviewers, providing resources and tools to ease their transition.

Feedback Loops: Enable contributors to provide feedback on the review process, creating a culture of continuous improvement.

Leverage Analytics for Continuous Improvement

Monitor DORA Metrics Regularly: Track key metrics continuously to detect trends and pinpoint areas needing optimization.

Set Performance Targets: Establish clear goals for deployment frequency, lead time, and other metrics to align the team toward common objectives.

Share Insights with the Community: Promote transparency by sharing performance data and achievements, fostering collective ownership of the project.

By adopting these strategies, OpenStreetMap can fine-tune its DevOps performance, further reduce lead times, and streamline workflows, strengthening its status as a leading open-source initiative.

Let's Wrap This Up

The OpenStreetMap website repository is a shining example of how effective DevOps practices can drive success in an open-source environment. With its impressive deployment frequency and smooth workflows, the project consistently delivers value to a global user base while maintaining high standards of reliability.

The strong engagement from contributors and maintainers creates a thriving community that continuously pushes innovation forward.

That said, there's always room to optimize. Focusing on cutting down lead times by implementing standardized response procedures and improving code quality could make the repository's delivery pipeline even faster. By incorporating these strategies, the project can enhance performance, boost contributor satisfaction, and ensure long-term sustainability.

Is Next.js the Next Evolution or Just a Passing Trend? A Dora Metrics Case Study

Shivam Chhuneja — Sat, 21 Sep 2024 05:46:10 +0000

Next.js is a repo that is a relatively recent yet powerful JavaScript framework that's taking the modern web development scene by storm. Next.js strengthens React with its server-side rendering, static site generation, and SEO optimization.

Developers love its versatility, and the numbers speak for themselves. In a recent survey, 60% of developers said they preferred Next.js for building production-ready applications due to its ease of deployment and scalability. When comparing deployment times, Next.js consistently outperforms React in speed and efficiency, making it a go-to choice for developers aiming for a seamless build process.

This case study focuses on the OpenSource Next.js repository, particularly highlighting its exceptional deployment frequency.

As developers who've tinkered with Next.js for building dynamic e-commerce sites and quick-launching blogs, we couldn't help but wonder---how did they create and manage such a legendary repo with such efficiency? So, we dug deeper using Middleware Open-Source.

If you're excited to explore these insights further and connect with fellow engineering leaders, come join us in The Middle Out Community or subscribe to our newsletter for exclusive case studies and more!

But, first things first. Let's understand what Dora Metrics are.

What is Dora Metrics?

Dora Metrics are key indicators that show how efficiently a project progresses from start to final production in software delivery. Feel free to read through our detailed blog on Dora Metrics and how they can help your engineering processes!

Deployment Frequency: How often code hits production.
Lead Time for Changes: How long it takes for a commit to go live.
Mean Time to Restore (MTTR): Time taken to recover from failure.
Change Failure Rate: How many of those deployments break something.

Now that we're all on the same page, let's see how Next.js measures up.

Our Key Findings

Next.js: Setting the Bar High with Exceptional Deployment Frequency

The OpenSource Next.js repository stands out for its exceptional deployment frequency, far surpassing industry standards. Over the past three months, the repo has consistently pushed a high volume of deployments, reflecting the team's efficiency and automation prowess.

In June 2024, there were 247 deployments, followed by 261 in July, and an impressive 279 in August. These numbers highlight the repository's commitment to continuous integration and rapid feature releases, making it a benchmark for deployment frequency in the open-source community.

That's nearly one deployment every 3 hours! This high-frequency deployment isn't just for show; it keeps the project evolving at an incredible speed. How do they do it? A couple of key factors stand out:

Automated CI/CD Pipeline: With automation in place, deployments are quick and smooth---almost like magic.
Frequent, Small PRs: By breaking down changes into bite-sized pull requests, they can merge and deploy faster with less risk.
Fast Issue Resolution: Bugs get squashed quickly, and failed deployments are rare.

Cycle Time leaves room for Improvement

Despite a high deployment frequency, the repository shows notable fluctuations in Cycle Time, which impacts the overall Lead Time for Changes. Cycle time includes all PRs, while Lead Time only includes PRs with available deployment data. In June 2024, the average Cycle Time was around 3.5 days, slightly increasing to 3.6 days in July, and then spiking to 5.3 days by August.

Cycle time has been on the rise, increasing from 3.5 to 5.3 days, indicating some delays in the development process. Identifying bottlenecks, whether in code reviews, testing, or manual tasks, and implementing automated testing or improved sprint planning could help reduce this.

Other Key Metrics that may use some work

First Response Time

In June 2024, the average first response time was 1.6 days (38.4 hours), indicating quick and efficient handling of issues and pull requests in the Next.js repository. By July 2024, this time increased slightly to 1.8 days (43.2 hours), hinting at a minor delay, likely due to a growing number of issues or reduced team availability.

However, in August 2024, there was a significant jump to 2.6 days (62.4 hours), suggesting noticeable delays, possibly caused by higher workloads, bottlenecks in reviewer assignments, or fewer team members due to vacations.

To improve First Response Time, several strategies can be implemented to address the recent increase in delays. Automating notifications for issues and pull requests can ensure faster assignment to reviewers, reducing wait times.

Introducing SLAs (Service Level Agreements) for response times could create accountability and encourage quicker engagement. Moreover, reviewing team allocation and workload distribution could help balance responsibilities, especially during periods of high demand or reduced team availability, such as vacations.

Encouraging cross-functional collaboration could also enable quicker responses by spreading the load across the team more evenly. These changes could effectively bring response times back down and maintain efficiency.

Lead Time for Changes

From June to August 2024, the lead time for changes in the Next.js repository gradually increased from 3.3 days to 4.1 days. In June, the relatively quick 3.3-day turnaround reflected an efficient workflow with smooth coordination. By July, this increased slightly to 3.6 days, hinting at minor inefficiencies, possibly due to longer code reviews or more complex tasks.

However, by August, the lead time reached 4.1 days, indicating more significant delays likely caused by higher workloads, bottlenecks in reviews, or reduced team capacity due to vacations. This trend suggests a need to optimize processes and prevent further slowdowns.

Reasons?

Extended review times. For instance, PR #67498
Complex tasks that involve deep scrutiny and testing. PR #67391
Fluctuations in First Response Times (from 1.6 days to 2.6 days) cause uneven start times for reviews, delaying progress.

Nature of Work

The OpenSource Next.js repo includes a variety of activities, ranging from feature upgrades and bug squashing to improving documentation and refining tests. Here's a breakdown of some key insights:

Features and Improvements: Overhauls including performance optimizations (PR #67397), new functionalities (PR #67215).
Documentation: Significant contributions to documentation ensure clarity and easier adoption (PR #67056).
Bug Fixes: Addressing critical bug fixes, highlighted by 41.5 hours (~1.7 days) of rework time (PR #67022).
Performance Optimizations: Enhancing the speed and efficiency of the framework (PR #67065).

These updates position Next.js as a leader in web development. However, many of the areas needing improvement---cycle time, first response, and lead time---can benefit from increased automation in testing, code reviews, and notifications.

To further optimize, they can track bottlenecks in their pipeline, whether these bottlenecks occur during code review, testing, or deployment, and target those areas to boost overall performance. Also, fostering continuous feedback through more frequent stand-ups or retrospectives can help identify and resolve friction points. By focusing on these strategies, you can create a more efficient and streamlined development and deployment pipeline.

How Does This Affect the Next.js Community?

For internal contributors, the rapid deployment cycle is a dream. Features and fixes roll out quickly, meaning the team gets feedback in near real-time. However, the high lead time could make some contributors feel like they're stuck in a long queue, waiting for their work to go live.

For external contributors, understanding these bottlenecks can set clearer expectations. If you're contributing to Next.js, don't be surprised if your PR takes a bit of time to merge, even though deployments happen frequently.

Key Takeaways

Invest in Automation: A solid CI/CD pipeline can keep deployments flowing like water.
Encourage Small, Frequent PRs: Less complexity means quicker reviews and faster deployments.
Address Bottlenecks in Cycle Time: If lead time is lagging behind, dig into what's causing the delays---whether it's rework or review times.

DORA Score: 8/10

Next.js has an impressive deployment frequency that's soaring, but their lead time, cycle time, and first response times could use some attention. With a few strategic tweaks, they have the potential to be strong contenders against the top performers in Google's Annual Dora report,

Is Next.js Making Other Open-Source Projects Obsolete?

Next.js has set a new benchmark in deployment efficiency, making it the one to watch in the open-source world. With its unbeatable blend of speed, quality, and constant innovation, it leaves little room for competitors to catch up. Whether you're an active contributor or a curious onlooker, there's plenty to admire---and learn---from this trailblazing repo.

As Next.js continues to evolve, it's not just keeping up with the web's demands; it's shaping them. Get ready for more groundbreaking updates as it redefines open-source excellence!

If you're excited to explore these insights further and connect with fellow engineering leaders, come join us in The Middle Out Community. and subscribe to the newsletter for exclusive case studies and more!

Trivia

Next.js was created by Vercel (formerly Zeit) in 2016 and quickly gained popularity due to its powerful features for building server-side rendered React applications. It supports both static site generation (SSG) and server-side rendering (SSR), making it a flexible choice for developers. Major companies like Netflix, Twitch, and GitHub use Next.js to power their web apps, showcasing its reliability and scalability.

Further Resources

Is FreeCAD Nailing PRs or Just Blazing Through Merges?

Shivam Chhuneja — Fri, 20 Sep 2024 05:04:59 +0000

FreeCAD, the open-source 3D CAD modeler, has garnered quite a community with over 12,000 commits, 1,100 contributors, and more than 5,000 pull requests (PRs) to date. Known for its versatility in mechanical engineering and product design, FreeCAD is a go-to for makers and professionals alike.

But are they actually nailing their PR process, or just racing through merges? With an average PR merge time of 7.5 days, FreeCAD's workflow efficiency is up for debate. Let's break down the numbers and see what's really happening behind the scenes.

We are using Middleware Open Source to get the data and insights for these open source repos, so if you want to recreate or get data of your favorite God level repo you know where to go!

If you're curious to start a conversation with fellow engineering leaders, join The Middle Out Community! Don't forget to subscribe to our newsletter below for exclusive case studies!

Channeling Our Inner Sam Spade: Cracking the Case of FreeCAD's PR Efficiency

As we at Middleware dug into FreeCAD's Dora metrics, we couldn't resist going full "Sam Spade," curious about what made this repository tick. So, we dove in to explore the patterns, looking for what makes them tick!. What we found sheds light on some best practices that other repos can take notes from.

Background on Dora Metrics

Dora Metrics are a set of key performance indicators used to measure software development velocity and efficiency. They include:

Deployment Frequency
Lead Time for Changes
Mean Time to Restore (MTTR)
Change Failure Rate

These metrics help organizations understand their development processes and identify areas for improvement.

Key Findings

Exemplary Metric: Merge Time

One standout metric for FreeCAD is its Merge Time for PRs. Efficient merge times are a testament to streamlined code reviews and strong team coordination.

June 2024 -- Average Merge Time: 15.66 Hours

In June, FreeCAD's PR management was running relatively smoothly, with an average merge time of 15.66 hours.

July 2024 -- Average Merge Time: 20.4 Hours

July showed a bit of a slowdown, with an average merge time stretching to 20.4 hours. This change could be attributed to more complex PRs

August 2024 -- Average Merge Time: 8.04 Hours

In August, things really picked up pace, with the average merge time dropping significantly to just 8.04 hours. This lightning-fast rate suggests that the team and community were hyper-efficient in handling pull requests.

Average Merge Time: Approximately 11.02 hours.

Rapid Merges: PRs like #14693, #14691, and #14688 were merged within less than a day.

Factors Contributing to Efficiency

Ownership: Consistent authors like PaddleStroke, Roy-043, and wwmayer contribute to swift merges.
Structured PR Review: Automated checks via GH Actions, including workflows like CI_master.yml and sub_lint.yml, catch issues early.

Also read: Is freeCodeCamp Sacrificing Quality for Speed with Their Rapid Deployments?

Metric that needs Improvement: Cycle Time

While freeCAD has made commendable strides in reducing Cycle Time, with figures decreasing from 171.12 hours to 130.8 hours and then to 121.68 hours, there's still room for improvement.

Continuing this trend could further enhance their efficiency. Addressing bottlenecks and optimizing workflows could help freeCAD push these numbers even lower, achieving even faster turnaround times and reinforcing their reputation for swift, effective development.

Average Cycle Time: Approximately 21.42 hours.
Fluctuation: Inconsistencies arise, as seen in longer cycle times for complex PRs like #14674.

Factors Contributing to Delays

Complex PRs: More intricate PRs understandably take longer.
Reviewer Availability: Delays due to the availability and assignment of reviewers.

Other Key Metrics

Lead Time

The reduction in Lead Time from 2.51 days in June to 1.81 days in August is a clear indicator of enhanced efficiency in moving code through the development pipeline. This is a commendable achievement, suggesting that freeCAD has improved its processes for deploying code, leading to faster delivery of new features and fixes. This efficiency boost likely contributes positively to the overall project velocity and responsiveness.

First Commit to Open

The increase in the time taken from First Commit to Opening a Pull Request, from 3.77 hours in June to 12.01 hours in August, is a bit concerning. This rise might indicate bottlenecks or delays in the initial stages of the contribution process. Addressing this could involve streamlining the process for handling code submissions, possibly by improving workflows or increasing the availability of reviewers to handle initial requests more promptly.

First Response Time

Although there's a slight improvement from 11.42 hours in June to 10.15 hours in August, there's still potential for faster response times. This metric is crucial for maintaining an active and engaged contributor base, as prompt responses can enhance collaboration and prevent contributors from losing interest or moving on to other projects. Focusing on reducing this time further could improve contributor satisfaction and keep the development process moving smoothly.

Nature of Work: Feature-Focused Development

FreeCAD is rich in diverse contributions, spanning feature development, bug fixes, and optimization work.

Feature Development

Major focus, e.g., Assembly: TNP by PaddleStroke.

Bug Fixes

Substantial effort in maintaining and fixing issues, e.g., Fem: Fix height of reference list widget in spring constraint task panel by marioalexis84.

Optimization and Refactor

Significant, seen in PRs like Guard all uses of basic_random_generator for thread safety by bgbsww.

Each of these contributions highlights the diverse nature of work being done on FreeCAD---from feature development that solves long-standing issues to bug fixes that refine the user experience, to optimizations that ensure the software can handle future demands.

Impact on Project and Community

Efficient PR management leads to better code quality, more frequent releases, and higher contributor satisfaction. FreeCAD's emphasis on swift PR reviews encourages further contributions.

Takeaways

Structured Workflows: Implement CI/CD pipelines to automate checks, as FreeCAD's CI_master.yml ensures consistency.
Active PR Review Culture: Prioritize swift reviews to avoid PR pile-ups.
Clear Ownership Assignments: Assign code owners to specific modules for better PR management.
Balanced Work Distribution: Encourage a mix of new features, bug fixes, and refactoring for a well-maintained codebase.

DORA Score: 8/10

After taking a closer look at FreeCAD with our Dora Metrics toolkit, here's the rundown: FreeCAD scores a respectable 8/10. Its PR merge times are impressively quick, thanks to a committed community that's always in motion. But---and there's always a "but"---Cycle Time is where FreeCAD hits a few speed bumps, with complex PRs dragging down the average.

The takeaway? FreeCAD's team is doing a solid job, but a little change here and there could take them from great to exceptional. Our analysis, stacked against Google's Dora benchmarks, highlights FreeCAD's wins and its opportunities for improvement. Want to see how your project compares or optimize for efficiency? Analyze it using Middleware OSS.

Conclusion - FreeCAD: A PR Powerhouse or Just Polished?

FreeCAD's got some slick moves when it comes to handling PRs---those rapid merge times are nothing short of impressive. But like a dance partner with two left feet, its Cycle Time could use a little more rhythm. With just a bit of fine-tuning, FreeCAD could be on its way to even greater efficiency.

If you're curious to pick the brains of fellow engineering leaders, join The Middle Out Community!

Trivia

FreeCAD was started in 2002 by Jürgen Riegel, Werner Mayer, and Yorik van Havre, making it one of the oldest free and open-source 3D CAD modeling tools

Further Resources

OpenVINO's AI Success: Brilliance or Cracks Beneath the Surface?

Shivam Chhuneja — Thu, 19 Sep 2024 04:37:38 +0000

Ever wonder how some AI projects manage to deliver breakthroughs in record time? OpenVINO, an open-source toolkit designed for optimizing and deploying AI models, is one such standout. With C++ at its core, this repository has consistently delivered on its promises. But what really sets it apart?

It's their ability to fine-tune and optimize Cycle Time---a critical metric that tracks how fast development work goes from inception to deployment. This case study takes a closer look at how OpenVINO manages to maintain an exceptional Cycle Time---showing us why this toolkit is making waves in the world of AI.

If you're excited to explore these insights further and connect with fellow engineering leaders, come join us in The Middle Out Community. and subscribe to the newsletter for exclusive case studies and more!

What We Discovered When We Peeked Into OpenVINO's Code Playground

Of course we love looking under the hood of how great teams and repositories function so we had to dive into OpenVINO as well leveraging Middleware OSS.

What made this repository tick?

Why was it performing so well in some areas, and what could be improved?

Using Dora metrics as our analytical framework, we dug into the data, analyzing everything from deployment frequencies to response times. What we found might just surprise you---and hopefully, inspire a few new ideas for your own processes.

Background on Dora Metrics

Before we get into the juicy details, let's take a moment to talk about Dora metrics. These are the gold standard for evaluating the performance and efficiency of software development and delivery pipelines. In simple terms, DORA metrics cover four key areas that provide a comprehensive picture of how well your dev processes are running:

Lead Time for Changes: How long does it take for a code change to go from development to production?
Deployment Frequency: How often are you pushing changes to production?
Mean Time to Restore (MTTR): How quickly can you recover from a failure?
Change Failure Rate: What percentage of your changes fail in production?

These metrics are crucial because they help assess the efficiency, speed, and reliability of your development pipeline.

For OpenVINO, we placed a special focus on Cycle Time, a close cousin to Lead Time, to see how efficiently they're moving code from start to finish.

Key Findings

Exceptional Cycle Time and Its Contributors

Cycle Time represents the time taken from when work begins until it's completed and ready for deployment. For OpenVINO, the Cycle Time over the last three months has been impressive. In June 2024, the First Response Time was 50 hours, dropping to 46 hours in July before rising to 68 hours in August.

These numbers highlight a consistent effort to keep development moving smoothly. But what exactly contributes to this exceptional Cycle Time?

Efficient Review Process: PRs like PR #25069 and PR #24953 were merged swiftly, with detailed and comprehensive reviews ensuring that quality wasn't sacrificed for speed.
Active Contributor Base: Regular contributions from a core group of contributors, like Ilya-lavreno and Tsavina have helped keep things on track. Active participation from dedicated contributors can significantly reduce bottlenecks in the pipeline.
Automation and Continuous Integration (CI): OpenVINO's robust CI setup plays a huge role here. They employ workflows like check_pr_commits.yml and build_doc.yml that streamline validation, testing, and integration. Automating repetitive tasks is a game-changer for any repository aiming to optimize its Cycle Time.

Also Read: JuliaLang: Performance Prowess or Just Smoke and Mirrors? Unveiling the Real Story

First Response Time: A Scope for Improvement

While Cycle Time has been a strong point, First Response Time---the time taken for maintainers to initially respond to new PRs---has room for improvement. Here's what the numbers say:

Although July showed a slight improvement, the overall response times remain high, with notable fluctuation. So, what's causing these delays?

Dependency on Key Contributors: Delays occur when critical reviewers are unavailable, which is common in any open-source project. The reliance on a small group of key contributors can slow things down.
Complexity of PRs: Some PRs---particularly those involving significant architectural changes or feature introductions---naturally require more time for initial assessment.

Nature of Work and Contributions

OpenVINO's recent work shows a diverse mix of feature development, documentation updates, and bug fixes. Here's a breakdown:

Some of the top features worked on recently include:

Implement Support for LLMs (PR #24930): Expanding the toolkit's capabilities with large language models.
Enable runtime buffer fusing for crop on GPU (PR #25060): Performance optimization for GPU-based workflows.
Activate SDPA Fusion on ARM (PR #24919): A critical feature for ARM architectures.

Impact on Project and Community

Internal Development Practices: The focus on efficient Cycle Times boosts overall productivity and ensures that development timelines are consistently met. This translates into more predictable releases and happier contributors.

External Contributors: Prospective contributors are more likely to engage with a repository that is both responsive and efficient. A streamlined development process not only enhances collaboration but also drives innovation by attracting new contributors who appreciate fast feedback cycles.

Takeaways

Effective Cycle Time Management

Strategy: Breaking down larger PRs into smaller, manageable pieces helps reduce overall Cycle Time and makes the review process easier.

Example: PR #25023 demonstrated effective segmentation, allowing for quicker reviews and fewer delays in merging.

Enhancing First Response Times

Strategy: Implementing a rotating schedule for first-level reviews can evenly distribute the workload among maintainers, reducing wait times.

Example: Introducing more automated triage processes could maintain response efficiency during high-activity periods, preventing backlogs.

Importance of Documentation and CI Pipelines

Strategy: Maintaining thorough documentation and robust CI pipelines helps streamline the review process and minimizes rework.

Example: Workflows like build_doc.yml have been key to keeping documentation up to date and comprehensive, which speeds up the development process.

Dora Score - 8/10

After diving into the Dora metrics for OpenVINO, we're giving it a solid 8/10. The project excels in Cycle Time and maintains a consistent deployment frequency, making it a standout in these areas. However, variations in First Response Time are holding it back from a perfect score.

When compared to Google's Annual Dora Report, OpenVINO holds up impressively, especially in terms of speed and efficiency. With some tweaks to improve those response times, it could easily leap into the elite tier.

Curious to see how your project stacks up? Middleware's OSS tools can help you dive into your metrics, find areas for improvement, and fast-track your project's success!

Also Check: The Dora Rating for another God-level Repo - React Native

Conclusion: Is OpenVINO's AI Speed Hiding a Need for More Precision?

OpenVINO knocks it out of the park with its impressive Cycle Time management, making its development process highly efficient and consistent. However, there's a bit of a snag---improving First Response Time could unlock even greater potential, speeding things up and driving more engagement from contributors. For teams looking to fine-tune their processes, OpenVINO's journey offers a great lesson in striking that perfect balance between speed and quality. If you are looking to bring more predictability & visibility to your engineering pipeline, then you should check out Middleware Open Source.

If you're excited to explore these insights further and connect with fellow engineering leaders, come join us in The Middle Out Community. Feel free to subscribe to the newsletter as well if you'd like some exclusive case studies too.

Let's Wrap This Up

OpenVINO was developed by Intel to enhance AI applications, particularly in edge computing, bringing AI processing closer to where the data is generated---think smart cameras, drones, and IoT devices.

Further Resources

Is Carbon Language Moving Towards Success or Self-Destruction? A Case Study in Rapid Deployments

Shivam Chhuneja — Wed, 18 Sep 2024 04:54:41 +0000

The Carbon Language repo on GitHub maintains an impressive deployment frequency, consistently pushing updates at a rapid pace. With an average of over 74.67 deployments per month---hitting 79 in July 2024 alone---Carbon is setting the pace for fast, efficient software delivery. But is there a price for going this fast?

This case study dives into the clever tactics that keep Carbon's wheels turning, while also uncovering whether Carbon's speed comes at a hidden cost. Buckle up---there's more beneath the hood than meets the eye.

If you're curious to connect with fellow software engineers, talk about all things DevOps, join The Middle Out Community!

Middleware OSS: Uncovering the Secrets Behind Carbon's Rapid Deployment

Middleware didn't just stumble upon the Carbon Language repo---we took one look at this high-speed project and knew it was time to roll up our sleeves. Middleware set out to examine how Carbon keeps its gears running so smoothly---and whether all that speed might be masking deeper performance issues.

With its aim to be a successor to C++, Carbon is fast, but can it balance that speed with long-term efficiency? This analysis sets the tone for the rest of the case study - a full diagnostic check. Middleware OSS is the tool for assessing what makes an open-source project tick---looking at deployment frequency, lead times, rework, and more.

Strap on the seat belt!

Oh, also this is one of the case studies in 100. If you want to get notified first and get exclusive access to more such case studies, make sure to subscribe to our newsletter!

Firstly, What are Dora Metrics?

Dora metrics is like the scorecard of your DevOps performance. It includes Deployment Frequency, Lead Time for Changes, Mean Time to Restore (MTTR), and Change Failure Rate. High-flying teams aim for the stars in deployment frequency while keeping lead time, MTTR, and change failure rates in check.

Key Findings

Deployment Frequency: The Star Metric

The Carbon Language repo is hitting home runs in deployment frequency, with 71 deployments in June 2024, 79 in July 2024, and 74 in August 2024.

Contributing Factors: What Fuels the Speed Machine?

Automated Nightly Builds

Imagine getting a new software update every night, great, right? Carbon's automated nightly builds are a perfectly planned system that ensures fresh code is deployed daily without a hitch.

These builds run quietly in the background, pulling all the latest changes together like clockwork, allowing developers to wake up to new features and fixes every day. It's like having a tech-savvy magician performing flawless deployments every night.

Continuous Integration Workflows

Think of CI workflows as the support team that operates behind the scenes, meticulously preparing everything before the main event goes live. They manage all the setup, perform every minute checks, and address any issues behind the scenes to ensure that the performance runs smoothly and seamlessly.

With GitHub Actions as the lead technician, they ensure that every piece of code gets its moment to shine, and every deployment rocks the house.

Efficient Task Segregation

When your tasks are well-organized, it's like having a perfectly labeled filing cabinet---everything's right where it should be. The Carbon repo has this down to an art. By keeping files and modules cleanly segregated, contributors can jump into specific tasks without hunting for the right pieces, reducing confusion and streamlining the entire process.

It's as simple as finding what you need with a glance, which helps keep productivity high and rework time low.

Also read: Quick Merge Times: Vue's Superpower

Areas of Improvement: Where the Speed Bumps Are

Rework Time

Rework time is a bit of a mixed bag for Carbon Language, averaging at 8.04 hours. The trend here suggests a growing need for refinement:

June 2024: A brisk 5.11 hours
July 2024: Slightly longer at 7.79 hours
August 2024: A notable increase to 11.23 hours

This upward trend points to increasing complexity or potential issues in the integration process. It's a signal that while Carbon is racing ahead with its deployments, there's room to fine-tune and smooth out the kinks. Time for the team to hit the workshop and give this engine a bit of extra TLC!

What's Slowing Them Down?

Complex Code Integration

Integrating new code into an existing system can sometimes feel like trying to fit a square peg into a round hole---awkward and challenging. When the code is complex, the process of merging changes can become cumbersome. This often involves dealing with intricate pull requests (PRs) that might need multiple revisions to align perfectly with the existing codebase.

It's like trying to assemble a jigsaw puzzle with missing pieces, and the result can be a bit of a tight squeeze. Check out these particularly tricky PRs #4095 and #4100 for a glimpse into the challenge.

Interdependencies

Imagine a game of Jenga, where each block represents a module or component in the codebase. Removing one block affects the stability of the entire tower, just as altering one module can impact the whole system. This delicate balance makes integration a complex and precarious process.

These interdependencies can stretch out rework time, as changes in one area often require adjustments in others. It's a bit like untangling a knot---you pull and tug, and it just seems to get more complicated the more you work at it.

Manual Intervention

Manual fixes are like speed bumps on a fast track. Every time a manual intervention is needed, it slows down the process and adds to the overall rework time. Instead of having automated systems handle repetitive tasks and fixes, these manual touch-ups can delay deployments and increase the chance of errors. It's time to introduce some automation wizardry to smooth out these bumps and speed up the process.

Nature of Work

Most contributions to the Carbon Language project are like intricate embroidery work. Each contribution adds a unique stitch to the overall design, requiring careful precision and attention to detail.

Just as a single misstep in embroidery can affect the final pattern, every contribution plays a crucial role in shaping the project's success. It's a meticulous process, combining intricate new features with precise documentation updates, thorough test cases, and careful bug fixes. The aim? To create a tool that runs as smoothly and reliably as a Swiss timepiece.

Contributor Details and More: Meet the Crew Behind the Code

Active Contributors

A dedicated team of 5 core members---including Jon Ross-Perkins, Chandler Carruth, Richard Smith, and David Blaikie---are driving this project forward. They're the ones steering this ship through turbulent waters.

High PR Activity

The pull requests are abuzz with activity. Each PR is a hive of energetic contributors, all bringing their best produce to the table. They're busy negotiating, refining, and perfecting the codebase, turning raw ideas into polished features.

These PRs ensure that every piece of code is thoroughly vetted and improved, keeping the project lively and up-to-date.

Amount of Rework

The project's complexity means an average rework time of 8.04 hours. It's like navigating through a dense forest---sometimes it takes a bit of extra time to clear the path.

Spotlight on the Latest Upgrades

Automated Nightly Builds

These are the nightly news broadcasts for the development world. With PRs 4055 and 4054 leading the charge, automated nightly builds have become a dependable part of the routine. These nightly builds ensure that fresh updates and improvements are delivered consistently, keeping the project on the cutting edge.

Hashtable Efficiency

PRs #4070 and #4046 have brought major enhancements to hashtable efficiency. Imagine upgrading from an old, rusty lock to a state-of-the-art security system. This optimization boosts data handling speeds and efficiency, ensuring that the system operates smoothly, handling data with precision and agility.

Generic Class Handling

The improvements in PRs 4086 and 4085 are akin to fine-tuning a high-performance engine. Enhanced generic class handling means code management is now more versatile and efficient. These upgrades make the codebase more adaptable and streamlined, resulting in a smoother development experience overall.

Impact on Project and Community

The secret behind maintaining high deployment frequencies? Automated builds! They're the game-changer that keeps things agile and ensures a steady stream of updates. Other open-source projects should take note and consider integrating similar automation strategies to stay competitive.

Takeaways

Leverage Automation

Integrating automated build scripts and workflows keeps the deployment frequency high. Aim for daily or weekly goals to keep the momentum going.

Modularize to Minimize Rework

Breaking down tasks into smaller, manageable chunks is like organizing your workspace. It makes finding and fixing issues quicker and easier. Review and structure your repo to support modular development.

Build Robust CI/CD Pipelines

Effective pipelines with automated tests ensure each release is top-notch. GitHub Actions can be your best ally in building and maintaining high-quality releases.

Dora Score: 8/10

After diving into the Dora metrics for Carbon Language, it's cruising with a solid 8/10. The project excels in deployment frequency and CI/CD pipeline efficiency, making it a top performer in those areas. But here's the twist---rework time is holding it back a bit.

When we stack this up against Google's Annual Dora Report, Carbon Language's performance is impressive, but a little fine-tuning on rework could propel it to the elite ranks. Want to see how your project measures up? Middleware's OSS lets you dive deep into your team's metrics, so you can pinpoint where to boost and get ahead of the pack!

Conclusion

Carbon Language repo stands out for its efficiency and innovation. Its remarkable ability to deliver updates at breakneck speed, thanks to savvy automation and meticulous task management, sets a gold standard for others to follow.

With a staggering average of over 74.67 deployments per month and nightly releases becoming routine, Carbon showcases how a perfectly managed development process can keep software fresh and functional.

But even top performers have room for improvement. While Carbon excels in deployment frequency and CI/CD pipeline efficiency, it could benefit from a tighter grip on rework time to reach its full potential.

Looking to scale your own project's performance? You found Middleware's OSS at the right time.

And if you're eager to dive deeper into these insights and connect with other engineering leaders, join us in The Middle Out Community.

Here's a fun trivia nugget about Carbon Language

Did You Know? Carbon Language, created by Google, was named after carbon, a fundamental element in chemistry and a building block of life. The name symbolizes the language's goal to be a foundational and transformative element in programming, much like carbon's role in nature.

Further Resources

JuliaLang: Performance Prowess or Just Smoke and Mirrors? Unveiling the Real Story

Shivam Chhuneja — Tue, 17 Sep 2024 05:23:52 +0000

Introduction

Julia, renowned for its speed and efficiency in scientific computing, has caught the eye of many in the data science world. We were eager to find out if there's real power behind the hype. Curious about whether JuliaLang lives up to its reputation as the sprinter of the programming world?

We were too! With the buzz around its agile performance and a huge open-source community cheering it on, we couldn't resist taking a closer look at the JuliaLang/Julia repository.

Armed with Middleware Open Source, we set out to uncover what makes this open-source project tick. You're in for some fascinating insights. Let's take a closer look.

If you're itching to spark some debates with fellow engineering leaders, dive into The Middle Out Community!

Background on Dora Metrics

Let's firstly understand what is Dora Metrics: DORA Metrics. These are like the ultimate report card for your DevOps efforts, showing off how well your team is doing. Here's the lowdown:

Lead Time for Changes: Time from code commit to production---how fast are you pushing those updates?
Deployment Frequency: How often are you rolling out new features or fixes?
Mean Time to Restore (MTTR): How quickly can you bounce back from a glitch?
Change Failure Rate: What's the percentage of your deployments that go sideways?

These metrics help us see what's working and what's not in our quest for top-notch software delivery.

Key Findings - Effective Code Quality Management

High-Flying Deployment Frequency

Let's talk numbers: The JuliaLang/Julia repository is rocking a consistently high deployment frequency. Over the last three months, their stats have been as follows:

June 2024: 122 deployments
July 2024: 164 deployments
August 2024: 179 deployments

That's right---these guys are cranking out updates like there's no tomorrow! By keeping deployments frequent and manageable, they minimize risk and keep the feedback loop running smoothly. The high deployment frequency indicates a highly efficient continuous integration (CI) process, enabling quick iteration and feedback loops. The repository achieves this by leveraging automated workflows and encouraging contributors to submit small, manageable changes.

Highlighted PRs:

PR #54882: Addressed diagonal matrix eigen decomposition, enhancing mathematical operations with precision.
PR #54864: Reverted recent changes to the garbage collector, restoring its previous configuration to maintain stability.

These incremental updates are a strategic choice, ensuring a smooth deployment process by avoiding large, disruptive changes and keeping the system running efficiently.

Also read: Swift Deployments: Are they Swift or Recklessly Rushed?

Areas for Improvement: Opportunities for Enhancement

Lead Time for Changes - A Bit of a Hiccup

Now, let's talk lead time---it's not exactly all smooth sailing here:

June 2024: 5.7 days
July 2024: 16.9 days
August 2024: 18.7 days

We've got some serious fluctuation going on here! The time it takes to get changes from commit to production vary, and it's something to keep an eye on. Here's what's causing the hiccups:

Review Time: Some PRs are like fine wine---they need a little time to age. Complex reviews, especially those requiring multiple perspectives, can slow things down. To streamline both review time, consider implementing a more structured review process that includes clear guidelines for complex PRs, allowing for quicker yet thorough evaluations.
Rework Time: With an average rework time of 1 day, it's clear that some adjustments are necessary, contributing to the overall lead time. Integrating automated code analysis tools can help catch issues early, reducing the need for extensive rework. Additionally, establishing a more efficient workflow for handling feedback and addressing rework promptly will mitigate delays. Regular training and clear documentation can also aid reviewers in providing quicker and more effective feedback, ultimately reducing both review and rework times and improving overall lead time.

These longer lead times can slow down the ability to rapidly iterate and respond to changes or issues.

Nature of Work and Contributor Insights

JuliaLang/Julia's repository is buzzing with contributions.

Feature Work

Numerous PRs are dedicated to introducing new features or enhancing the performance of existing ones, driving continuous improvement.

Documentation

Keeping user-facing documentation up-to-date with clear explanations of new features and changes ensures accessibility and ease of use.

Bug Fixes

Focusing on resolving issues and improving system stability, making the overall experience more reliable.

Highlighted PRs

PR #54812: Fixed lowering for export---because precision matters.
PR #54446: Added syntax highlighting to Exprs in REPL---making the code pop in the terminal like never before.

Impact on Project and Community

These findings shape how both the internal team and external contributors experience the repository. High deployment frequency keeps things lively, while addressing lead time issues will enhance overall agility and responsiveness.

Takeaways

Frequent Deployments

Aim for smaller, incremental changes rather than larger, riskier updates. This strategy not only keeps the deployment frequency high but also minimizes the potential for large-scale issues, making it easier to spot and fix problems quickly.

Effective Use of CI/CD

Maximize the efficiency of your CI/CD pipeline by automating workflows. Continuous integration and deployment will streamline the process, allowing changes to be merged and deployed swiftly, without bottlenecks.

Improving Lead Time

To cut down on review and rework times, consider increasing the pool of active reviewers. This, combined with more thorough testing during the early stages, can provide faster and more actionable feedback, accelerating the process from code submission to deployment.

Collaborative Contributions

Promote a culture of collaboration where contributors feel empowered to participate in feature development, bug fixes, and documentation improvements. By creating an open, engaging environment, you can enhance both the quality of contributions and the overall pace of development.

DORA Score: 8/10

After sleuthing through the Dora metrics for Julia's repository, it's earned a strong 8/10. The frequent deployments and commitment to top-notch quality are definitely standout performers. But here's the thing---lead time for changes could use a little tightening up. Fix that, and we're talking near perfection!

Comparing this to Google's annual Dora report, Julia's repo holds its own against the big players, but with just a bit more polish on lead time, it could jump to the top ranks. And guess what? And if you wish to track your project's movement then Middleware's OSS is your go-to tool.

Also read: Did React's Repo Miss The Speeding Sign? A Look Into Their Speedy Features & Bug Fixing Delays

Conclusion: The JuliaLang Jigsaw---What's the Real Deal?

Here's the assessment of the JuliaLang/Julia repository: It's a strong performer, delivering impressive deployment frequency, but the lead time for changes could benefit from some refinement. With a solid score of 8/10, JuliaLang is operating efficiently, showing swift and frequent deployments, thanks to a highly capable team of developers keeping things running smoothly. However, the occasional delays, like longer lead times, do present opportunities for improvement.

The JuliaLang community can smooth out these minor inefficiencies to further enhance performance. For the rest of us, there's a lesson in maintaining high-frequency deployments while keeping an eye on areas that might need adjustments.

If you're itching to spark some debates with fellow engineering leaders, dive into The Middle Out Community!

Trivia

Julia is known for being ridiculously fast. It's compiled using LLVM (Low-Level Virtual Machine), meaning it can execute code almost as quickly as C, making it a favorite among data scientists who don't have time for slow computation

Further Resources

React Native's CI/CD Unveiled: The Truth Behind Its Cycle Time Triumphs and Stumbles

Shivam Chhuneja — Mon, 16 Sep 2024 10:31:00 +0000

Introduction

React Native's journey from a humble internal Facebook tool to a global mobile development powerhouse is remarkable. But behind this open-source star's success lies a more intricate story --- one of productivity, efficiency, and a few bumps along the way. With its complex blend of JavaScript, C++ scripts, and cutting-edge CI/CD workflows, React Native's repository is more than just code; it's a living testament to modern development practices.

If you're excited to explore and connect with fellow engineering leaders, come join us in The Middle Out Community. Let's keep the dialogue dynamic and the innovations unstoppable!

Middleware Investigates: The React Native Productivity Puzzle

Middleware took a keen interest in the React Native repository. Intrigued by its rapid global adoption and the buzz surrounding its development practices, Middleware decided to dive deep and scrutinize the team's productivity.

Using Dora metrics as our detective lens, we set out to understand the workings behind React Native's impressive cycle times and the occasional obstacles in their workflow.

Without further delay, let's dive into what React Native is excelling at and where it's hitting roadblocks

Also Read: Did React's Repo Miss The Speeding Sign? A Look Into Their Speedy Features & Bug Fixing Delays

Dora Metrics Overview

DORA metrics are like the ultimate scorecard for assessing software development and delivery performance. They include:

Lead Time for Changes: The duration from code commit to its live debut in production.
Deployment Frequency: How often the team manages to release updates to production.
Mean Time to Restore (MTTR): The time required to get services back on track after an incident or outage.
Change Failure Rate: The percentage of changes that either degrade service or need fixing.

These metrics are crucial for spotting bottlenecks and evaluating the overall efficiency and quality of the development and deployment processes.

Key Findings

The Power of React Native: Mastering Cycle Time

The React Native repository shines in the realm of Cycle Time. Between June and August, the team consistently delivered impressive performances.

The average Cycle Time reveals that the team is capable of pushing features and fixes into production in under three days. Noteworthy PRs like #46066 and #45840 exemplify this swift cycle time.

This efficiency is driven by robust CI/CD practices, automated testing, and quick code reviews. Tools like autorebase.yml and test-all.yml highlight the streamlined approach, trimming the time from development to deployment.

Another Big Win: Quick First Response Time

The repository shines when it comes to First Response Time. It's a key metric that measures how quickly reviewers jump in and start engaging with new pull requests. Essentially, it tracks the speed at which someone takes the first step to review and provide feedback, helping to keep the workflow moving smoothly and reducing delays in the development process. This quick engagement boosts team productivity and ensures that issues or improvements are addressed promptly.

Typically, the repository manages to respond to pull requests within 12 hours, which is vital for maintaining a brisk development pace.

Particularly in June, the average first response time was impressively quick at 1.71 hours. This swift feedback loop allows contributors to quickly address issues or make adjustments.

However, July saw a notable spike to 16.69 hours, possibly due to varying reviewer availability or pull request complexity. Despite this, the overall trend shows a strong commitment to timely communication.

Trouble Spot: Merge Time

On the downside, Merge Time has been a concern, especially in August.

In the given chart, the sharp rise in August indicates possible issues such as complex changes, a higher volume of pull requests, or reduced reviewer availability.

For instance, PR #45894 experienced delays, likely due to review wait times or complex post-merge tasks.

Nature of Work and Contribution

Most of the action in this repository revolves around leveling up features, fine-tuning CI/CD pipelines, and squashing bugs. Here are some of the recent standout updates:

TurboModules Support: This includes a more efficient way to handle com.facebook.react.bridge.Dynamic, making module handling smoother and more robust.
Hermes Engine Updates: The Hermes engine received some key tweaks, particularly around efficient slicing, which helps improve performance and reduce overhead when handling JavaScript execution.
Test Case Enhancements: Both end-to-end and unit tests saw some major upgrades, with enhanced scripts that ensure more reliable test results, helping to catch bugs faster and keep the codebase stable.

All these improvements keep the repository in top shape and moving towards better efficiency.

Key Metrics Overview

Features and Tasks

The repository covers a wide range of tasks, from fine-tuning core functionality to tweaking CI configurations. It's like a toolbox for everything, ensuring both the essential features and the behind-the-scenes processes are kept in check.

Contributors

Key contributors like Shell, Cortinico, and Cipolleschi have been driving the project forward. Their consistent contributions showcase not only technical expertise but a strong commitment to improving the repository.

They're the driving force behind the recent progress. Their expertise and dedication are crucial in pushing the project forward, making sure that new features are developed, bugs are fixed, and improvements are continuously made. Without their contributions, the steady flow of enhancements and updates we're seeing wouldn't be possible.

In essence, they're the key players ensuring that the project not only progresses but thrives.

Rework

There's been minimal rework, which speaks volumes about the stability of the initial implementations. Things are being built right the first time around, minimizing the need for revisiting and fixing past work.

Deployment Frequency

With 32 deployments between June and August, this repository is moving fast. The high deployment rate highlights a dynamic and active development cycle, keeping new features and fixes rolling out steadily. It's a clear sign of a constantly evolving and improving project.

Impact on Project and Community

These findings are valuable for both internal and external contributors. The quick Cycle Times foster rapid feedback loops, crucial for open-source projects with frequent community contributions. Meanwhile, the delays in Merge Time underscore the need to balance speedy code reviews and maintaining high code quality.

Takeaways

For those keeping track, here are some essential takeaways:

Automate CI/CD Pipelines

Think of it like setting your favorite playlist to shuffle---once it's set up, you don't have to worry about it, and it keeps the party going smoothly. The React Native team's knack for automation in their CI/CD pipelines is key to keeping their development cycle quick and efficient. So, invest in solid automation practices to keep things running like a well-oiled machine.

Prompt Code Reviews

Quick reviews are the magic required to keep the development process on track. Even though there might be the occasional hiccup, getting those reviews done quickly helps avoid bottlenecks and keeps everything moving smoothly.

Effective Communication

Clear PR descriptions and regular discussions are like having a GPS for your project---guiding everyone in the right direction and avoiding unnecessary detours. This minimizes the need for rework and helps prevent delays, keeping everything on track.

For a taste of what efficient practices look like, check out PR #45062 and PR #44794. They're prime examples of how to do things right.

Dora Score

Rating: 8/10

After deep diving into the Dora metrics for React Native, we're giving it a solid 8/10. The project excels in cycle time and quick first response time, making it a real standout in these areas.

However, there's a minor hiccup---merge times are causing some delays that slightly impact the overall score.

When we compare React Native's performance to Google's annual Dora report, it holds up impressively. With a few tweaks to those merge times, it could easily leap to the top tier.

If you are also curious to find how your project measures up? Middleware's OSS has got you covered. It's packed with tools to dive deep into your team's metrics, spot areas for improvement, and put your project on the fast track to success!

Also: Did you check our Dora rating for Ruby repository?

Conclusion: React Native's CI/CD Hype: Are We Celebrating Efficiency or Ignoring Key Bottlenecks?

The React Native repository stands as a great example of how efficient development cycles can be achieved with a strong foundation of robust CI/CD practices. This case study illustrates the transformative power of continuous integration and continuous delivery, which not only accelerated development but also ensured consistent code quality.

Despite the successes, there's still some room to finesse the merge processes. Streamlining these workflows could further enhance efficiency, reduce bottlenecks, and improve collaboration among team members. Yet, the project's current state exemplifies a commendable balance between rapid iteration and rigorous quality control.

In essence, this repository showcases that with the right tools and practices, it's entirely possible to keep the pace brisk while ensuring that the end product is polished and reliable. It's a testament to the fact that maintaining high standards doesn't have to slow you down---instead, it can propel you forward with greater confidence and speed.

As we continue to refine and enhance our processes, the React Native repo remains a valuable blueprint for achieving a harmonious blend of agility and excellence in software development.

If you're excited to explore these insights further and connect with fellow engineering leaders, come join us in The Middle Out Community. Let's keep the dialogue dynamic and the innovations unstoppable!

Trivia

React Native was originally created by Facebook as an internal tool to solve a problem: how to develop mobile apps faster and more efficiently. But here's the twist---its creation was inspired by a heated debate about the lack of native performance in web apps.

Fast forward to today, and React Native has become a superhero of the mobile development world, allowing developers to craft high-performance apps using JavaScript and React. Who knew a coding argument would lead to such a game-changer?

DEV Community: Middleware

The 2024 DORA Report: State of DevOps Breakdown Summary

1. State of Software Delivery Performance

2. AI's Role in Software Delivery

3. Platform Engineering's Impact on Developer Experience

4. Developer Independence and Self-Service Workflows

5. The Role of Documentation in Developer Productivity

6. User-Centric Focus and Transformational Leadership

Let's Wrap This

Electron Repo—Unpredictable Metrics, Efficient Deployments

Understanding Electron Dora Metrics: The Domino Effect

Electron Engineering Pipeline: Busy Putting out fires

The Key Strengths They Can Cash In On

Leaving Thoughts: Electron Repo---Unpredictable Metrics, Efficient Deployments

Did you know?

TypeScript Dora Metrics Analysis: They Just Need a Smart Game Plan!

1. Deployment Frequency, Merge Time, First Response Time & Rework Time Analytics

2. Their Strengths: A Powerful Ecosystem and A Solid Backing

3. Leveraging Strengths to Build a Robust Engineering Roadmap

Capitalize on Active Contributions

Bug Fixes: Taming the Beast with Quality Code and Smart Strategies

Strategies to Resolve Delays in Feature Pushes

Our Take: TypeScript Has the Tools to Build a Winning Roadmap!

Before you go, here's TypeScript Trivia!

Flutter’s Engineering Pipeline: A Great Workflow with Minor Hiccups

Flutter's Engineering Pipeline: Robust Yet Vulnerable

Flutter's Strengths and Best Practices

Code Reusability: Write Once, Use Everywhere

CI/CD Made Easy: Smooth Sailing for Releases

Frequent, Low-Risk Changes

Active Community Engagement

Steady Deployment Cadence

Improving Performance: How Flutter Can Prevent Future Anomalies Like July's Dip?

1. Backup Reviewers During Community Downtimes

Automate Workflows, Distribute Complex Reviews

3. Host Hackathons; Recognize Contributors to Boost Community Engagement

4. Make CI/CD Pipeline Efficient

Dora Metrics Rating

Flutter's Engineering Pipeline: A Great Workflow with Minor Hiccups

Trivia

Bootstrap: Strong Merge and Cycle Times, but First Response Time Needs a Revamp

Bootstrap's Key Strengths and Paths for Improvement

Areas Where Bootstrap Shines

Area for Improvement: Enhancing First Response Time

Our Verdict

Bootstrap's Strength

Leveraging Strengths to Overcome Weaknesses

Conclusion: Bootstrap Excels in Merge and Cycle Time, But First Response Time Remains a Major Weak Spot

Trivia

Further Resources

Optimizing OpenStreetMap’s DevOps Efficiency: A Data-Driven DORA Metrics Analysis

Understanding DORA Metrics

Key Findings

1. Deployment Frequency

Key Drivers of Deployment Frequency

2. Lead Time for Changes

Key Influencing Factors

Diverse Contributions Pushing Growth

Positive Impact on the Project

Strategic Recommendations for Enhanced Performance

Standardize and Expedite First Response Times

Reduce Rework Through Enhanced Code Quality

Foster and Sustain Active Code Review Culture

Leverage Analytics for Continuous Improvement

Let's Wrap This Up

Further Reading and Resources

Is Next.js the Next Evolution or Just a Passing Trend? A Dora Metrics Case Study

What is Dora Metrics?

Our Key Findings

Next.js: Setting the Bar High with Exceptional Deployment Frequency

Cycle Time leaves room for Improvement

Other Key Metrics that may use some work

First Response Time

Lead Time for Changes

Nature of Work

How Does This Affect the Next.js Community?

Key Takeaways

DORA Score: 8/10

Is Next.js Making Other Open-Source Projects Obsolete?

Trivia