Categories
Tips

Common developer frustrations and how to overcome them

Derek Lee Boire, Senior Member of Technical Staff at VMware Pivotal Labs, addresses common developer frustrations, such as decision fatigue,  and the tools and methods to overcome them.

If you’re a developer, whether you realize it or not, you’ve probably experienced ‘decision fatigue’ at one point in your career. Decision fatigue occurs when the quality of the decisions we make deteriorates due to the mental fatigue of having already made a lot of decisions. While some people may want to believe that they have infinite willpower, our mental energy and focus has a limit no different than our physical bodies do. You can train long, or you can train hard – either way your body will tire eventually.

But just as we can train our bodies to endure additional stress, we can also train our minds to increase our focus. Or ideally – simplify our work to allow us to save our energy for the mental challenges that really make a difference. For example, a person with a clear idea of how they would like to manage their diet will be able to make easier decisions when grocery shopping, which can be helpful when different options are particularly abundant.  

While this example refers to a relatively simple, personal decision, it’s even more important to alleviate the burden of choice for more complex decisions which involve a greater number of stakeholders. These are the types of decisions software developers will make with each line of code written, putting them at a high-risk of experiencing decision fatigue. Here we look at some of the ways in which decision fatigue and other common developer frustrations that can manifest in software development, and the tools and methods that can be employed to combat them.

Common developer frustrations

Discoverability

If there is a task that must be performed many times, or takes a long time to do manually, or is complicated and error-prone, developers should consider using automation. Continuous integration and continuous delivery (CI/CD), the practice of automating testing and deployment of your application, is one example of this. When you automate a process, you don’t just make it easier to execute, save time, and ensure consistency – you also make it discoverable. 

This means that if a new member joins the team and has never worked with CI/CD before, reviewing the script that manages this automated process will help them to easily understand how the application is deployed. 

This can apply to source-code as well. If you’ve ever worked on a backend API that is consumed by others, you know how important it is for your API to be easily understood in order for users to easily consume it. This same level of understanding can be applied to your internal APIs that you build for yourself, for future you, and for other members of your team. 

As software engineers, we are all authors. As such, we need to consider the audience who is consuming the code that we are writing. 

Rabbit Holes

When working independently, it can be easy to find yourself lost down a “rabbit hole” – deep into trying to solve a problem and unable to remember where you had started or why you went down this route in the first place. Developers working in isolation are particularly prone to finding themselves in a rabbit hole, which is a sure-fire sign that decision fatigue is settling in. 

Collaborative development processes, like pair programming, help developers avoid rabbit holes through creating a culture of fast feedback and knowledge-sharing. When there’s only one person who understands how a specific feature was implemented, this carries a risk for the team if that person is out on vacation, happens to get sick, or is out for any number of reasons, then the team can get stuck. 

One recommended practice for pair programming is to rotate who pairs with who daily to ensure that several different team members have knowledge of how each part of the system works. This is a great way to share knowledge throughout the team as well as to ensure that the team can always continue to make forward progress, even if a team member is unavailable. 

Pair programming is built upon equality and mutual respect, a by-product of which is the establishment and/or sustenance of a culture of knowledge-sharing. Approaches to collaboration based on equal parts observation and communication are always more effective in keeping people engaged. After all, too much of one or not enough of the other can make it difficult to concentrate, which is especially detrimental when trying to absorb or share knowledge.  For example, I recently learned from a pair programming session about how test-driven deployment (TDD) could be applied to configuring docker containers using scripting purely vicariously.

Domino Code

Unsure how to implement a new feature or enhance an existing feature? If there’s a similar feature in the application, it could be a good reference so you can just follow the existing code patterns and tack on the new functionality, right? Sure, this might work for the first or second time – but by the third or fourth time you’ve repeated this pattern, it could start to get unwieldy. The easy way may be to follow the same pattern, and this can be good to maintain consistency in the codebase. However, to ensure that the software can continue to evolve, a different abstraction might be helpful. 

Developers often see an existing pattern and simply copy and paste or tack on more of the same.  While the pattern may have been easy to understand when it was initially implemented, repeated use can eventually create a ‘domino’ effect –  whereby that same pattern, at scale, may no longer serve the code after it has evolved. 

While there are some developers who are strictly against code duplication, it can sometimes actually be helpful. There’s a delicate balance between code duplication and creating a premature abstraction. Sometimes duplication is, in fact, necessary if an abstraction or cleaner way of writing can’t yet be discerned. It can be good to just let the code “simmer” for a bit longer and see how it will evolve. 

Developers should therefore be aware of the code being copied or the patterns continually being promoted in the code. The next time you find an existing code pattern and your initial instinct is to simply copy it, consider this: the next time you or someone else encounters this code, would it make sense to continue using this pattern, or is it time to consider refactoring or creating an abstraction? A good example is when a class becomes so long that it is difficult to manage (also known as ‘long class code smell’ amongst developers). The easy action is to continue to add to it and make it longer. When you detect this “smell”, consider if it would become easier to manage if it was broken down into more than one class. Are there multiple responsibilities within the same class that could be separated?

Culture of Growing and Sharing Knowledge

This is by no means an exhaustive list of all practices for building code which works for everyone. Consistency in the practices deployed (i.e. code style, formatting, automation) is another significant area where many of the major barriers can be overcome. But the first step toward implementing any of the practices described here involves a more widespread, cultural change within the organisation itself. 

Developers work best when organisational culture promotes growing and sharing knowledge within the team. This should be approached comprehensively rather than on the basis of incentives: “lunch and learn” sessions are great, but free food will only get you so far when you don’t really get to take a break from work. Instead, development teams should tie core values like communication, courage and respect to every aspect of the work they do. Doing this will help alleviate common barriers faced by developers to writing great software, while also contributing to an increase in productivity, quality of work, and developer well-being.

About

Derek Lee Boire is a Senior Member of Technical Staff at VMware Pivotal Labs. Derek’s career as an IT professional and consultant has given him experience in every aspect of software development. From working with clients to understand and detail their requirements, prototyping solutions, designing technical architectures for large-scale systems, leading development teams and helping implement agile design methodologies. 

GitHub: derekleerock

Twitter: @derekleerock

Further reading

We cover Dev Ops CI/CD usage trends here. Interested in insights? Check out our latest research reports and graphs based on data from developers like you who took our global surveys.

Categories
Tools

A Guide to Choose the Right JavaScript Framework for Web Programming

The platforms of JavaScript probably are the best frameworks for developers as well as business organizations nowadays. Already, there is a vast ocean of JavaScript frameworks, with new ones pouring in constantly.  Frontend developers have difficulty in making up their minds as to which framework to stick to, particularly for single-page apps.

There are so many JavaScript frameworks and libraries to choose from on the web. Among the most popular however include React, Angular, Ember, Meteor, and Vue. The frameworks of JavaScript are preferred over others due to their functionality and interoperability. The libraries offer a better functional approach, good scalability, and most important of all platform independence.

JavaScript at present is used on the client and server-sides. The platform helps in designing an enriched user experience with many features to alter web pages in real-time. For fast web application development, JavaScript is the most preferred and acts as the perfect skeleton for single page applications, enabling developers complete focus on the interface elements, expanding the platform capabilities.

To further determine the best JavaScript framework to choose that best fits your web programming requirements, take a closer look at the five best JavaScript frameworks. 

Five best JavaScript Frameworks


Which JavaScript frameworks are you using? Let us know in our latest Developer Economics survey.

React JavaScript Framework


JavaScript frameworks - React

The brain-child of Facebook: it’s the framework in which the highest viewed social media platform, Facebook is based on. It’s supported by the FB team and also the driving force behind Instagram. It’s the choice of developers for many reasons to create dynamic and high-traffic web apps. 

Deemed as the quickest growing framework, with over 1000 GitHub contributors which increases the use of ReactJS in development with high demand.

React acts as a View in the MVC framework, and could be integrated seamlessly and smoothly into any architecture. One component file has the HTML markup and the business logic. You can either use Flux or a similar JS library for components communication. 

The seamless transfer of data between components is easy with the help of the props objects and the state. Credits to the Virtual DOM and special reconciliation algorithm, front-end gets a considerable boost. Changes could be done by developers in real-time, without having to update the View each time.

Pros:

The major component of boosting performance is the virtual DOM. Since React is framework-friendly, you could integrate it with other frameworks for inflated compatibility and performance. Moreover, it could be utilized on the client and the server-side. 

Cons:

Its flexibility is also its downside. It’s not easy to choose from the overabundance of frameworks. It works only on the MVC Framework’s View layer. 

In spite of the inclusion of the major code libraries, in some instances it still needs more code. And since it’s more of a library instead of framework, data changes need manual processing. 

Angular JavaScript Framework


JavaScript frameworks - Angular

The web-app platform is type-script based, making it exceptionally agile. Led by the Angular team at Google, a community of members and testers, it’s an open-source solution. Referred often as an MVW framework, it’s considered the best to use, particularly among startups and medium-size enterprises.

What sets Angular apart from the rest of the JavaScript frameworks is its two-way data binding, which makes backend changes reflect in real time on the user interface. As the most sought-after apps for Single Page Applications, an Angular JS development company enjoys high demand. 

It’s preferred for hybrid mobile apps as well. Features like switches, sidebars, and overlays integrate dynamic functionality to apps. It came about in 2016 with the version 2.0, the immediate release occurred in Angular 4 in the following year, with updates at regular intervals. Angular 5 was released in 2017 followed by 6 in the middle of 2018, then Angular 7 in September of the same year.



The diagram of the architecture identifies eight major building blocks of Angular 2 app.

Components – are a logical code for a JS app, consisting of a class, template, and metadata. 

Modules – used in creating a particular set of instructions. Rather than coding everything in a single place, you could build individual modules, which depend on the functionality. A module contains an export array, bootstrap array, and an import array as well.

Metadata – has the extra data defined for classes and is defined with a decorator.

Data binding – used for binding classes properties in the components.

Directives – a custom element of HTML. Used for extending the HTML functionality, with two directives as part of the Browser Module module.

Services – service is built to provide common functionality across various program modules. 

Templates – used for rendering the View for an app, consisting of the HTML code required for binding, rendering, and directives.

Dependency injection – used for extending module functionality during runtime.

Pros:

Yields huge performances, which is made possible by type-scripting. It offers libraries and functionalities for nested components, as well as web components support.

Cons:

It modifies a pre-existing DOM, which makes it slow and inefficient compared to React. It maintains a physical distance between the JavaScript driving an app and the HTML rendered.

Vue.js JavaScript Framework


JavavScript frameworks - Vue.js

Was introduced in 2016. It instilled the better parts of React, Angular, and Ember and put into a handy package. Vue’s learning curve was better than Angular and React.

What makes it stand out among the rest is the two-way binding, optional JSX support, Vue-cli, server-side rendering, and more. The ecosystem of the framework is composed of a view-loader as a component, library, Chrome, Vue.js devtools, Firefox, as well as a dedicated library Vuex to manage app state with the framework. Thus, it’s the better option than its counterparts for SPAs and cross-platform solutions.

You might see the Vue library as a combination of React and Angular, and somehow it’s true. It borrows the concepts of both React and Angular concepts. Layouts and logic, together with stylesheets are stored in a file. It also utilizes props objects and states, the same as React.

A reason to opt for Vue over React nonetheless is that React apps used the Redux library. As the app grows in size, building mini changes across files has the tendency to be tedious. This makes the Vue framework more favourable than others.

Pros

A collection of the best parts of the majority of frameworks. Thus, understanding the ease and deployment comes around. Very flexible, as well as facilitates a two-way communication, with a virtual DOM as well.

Cons

The downfall with Vue is its high flexibility. The high integrity makes choosing one for development hard. Most users of the Vue.js framework belong to non-native English speakers’ closed communities. The barrier in language is one of the reasons why developers are wary of using it.

React Native JavaScript Framework


JavaScript frameworks - React Native

It was a breakthrough when it comes to mobile application development. Before React, developing an Android and iOS app required various code sets. The whole deployment and coding functionality varied, and took a big toll on the expenses. 

React Native was launched by Facebook a few years back, revolutionizing the phone apps concept. You could write an app code and deploy at the same time on iOS and Android. The great success was the write once, deploy anywhere concept and React since then has been the foremost choice of native application development.

The framework takes the development of mobile apps to an all-new level, using CSS-like stylesheets, JS code, and HTML tags for lay-outing.

Pros

The concept of one framework, numerous platforms is a great React Native bonus. It has an extremely streamlined UI, and you could make apps faster with them.

Cons

The navigation is a turnoff, and transitions are not smooth either. It lacks the required custom modules for native application development. Native developers moreover are still required to accomplish the task.

Ionic JavaScript Framework


JavaScript frameworks - Ionic

It’s all about performance and nothing else. A cross-platform, and open-sourced, it’s used for developing hybrid mobile apps. A mobile HTML5 framework, it’s solely aimed at performance, using hardware acceleration. What’s great is that it does not require a third-party JS library. 

Ionic could be teamed up with Angular.js for building an interactive application. It comes with a set of its own icon set, called the Ionicons. Then it’s teamed up with reusable HTML classes for building an interactive mobile user interface. The HTML5 framework works with the help of Cordova or PhoneGap. 

Pros

A once code for all Ionic2 Concepts makes it favourable for developing an app.  Moreover, it’s open source and thus free entirely. The open source advantage of the huge developer community, as well as testers ascertain that the framework is top notch at all times. The open source benefit of a big community of developers and testers ensures that Ionic is on the top always.

Cons

Because of the code simplicity, creating an in-app navigation could be burdensome. It’s a very complicated task. The performance of the in-app is swift as the app when developed for individual devices. Because Angular.js is a must for Ionic 2, Angular.js knowledge is a critical factor in app development.

JavaScript is highly favoured among all the other programming languages. A few benefits are the following.

1. Safety. They are in continuous iteration, therefore they have a large community of members, as well as testers. 

2. Expense. Web applications cost an arm and a leg as most frameworks cost more. Most of the frameworks of JavaScript however are free of cost. This lowers the costs of web development drastically.

3. Efficiency. As earlier mentioned, the JavaScript frameworks are cutters of cost. In addition, they are also time savers, credits to the prebuilt functionalities for a faster web application development.

Conclusion

The whole website development community is extremely agile and continuously updates. Every month, a new framework is being released.  Those that are released are under constant upgrades and enhanced with the latest functions. We hope this guide helps you to choose the right JavaScript framework for you.

Author Bio:

Olivia D. is a technology observer, writer and blogger. Being a tech geek, she keeps a close watch over the industry focusing on the latest technology news and gadgets. Follow me on Twitter.

Are you using a JavaScript framework? Let us know in our latest Developer Economics survey.


Be a guest writer on our blog
Have you got brilliant tips and resources that developers love to read? Then we want you on our blog! Find out more.

Categories
Tips

Scale Up Deep Learning in the Cloud

Deep learning is typically a long and costly endeavour, especially when it comes to training models. There are many factors that impact the process, but processing power, in particular, can make or break your pipeline. Today, many developers leverage graphics processing units (GPUs). Learn how you can scale up deep learning in the cloud. 

GPUs enable you to run simultaneous compute operations. This capability can significantly speed up your model training time. While on-premise GPUs aren’t an option for everyone, there is an increasing number of cloud-based GPUs options you can take advantage of.

Who Is Using Deep Learning in the Cloud?

As of Q4 2019, there are 13.3M developers working on data science, machine learning and AI development worldwide, up from 12.2M a year ago,  based on the findings from the Developer Economics Q4 2019 survey.

Click here to help us update the figure for 2020 – take part in our latest Developer Economics Q2 2020 survey live now.


Developer Economics survey - speak out and win prizes.

While many of these developers are working on more accessible, and budget friendly, machine learning (ML) projects, deep learning implementations are also gaining traction. The obvious organizations to look to for this are the cloud providers themselves and other industry giants, including Google, Facebook, and Microsoft. Others include Snapchat, Fermilab, Disney, and Carnegie Mellon University.

However, deep learning in the cloud is also proving beneficial for many smaller organizations that would otherwise not have access to the technology. As larger organizations have increased their adoption, the breadth and availability of services has increased and the cost has gone down. This has paved the way for deep learning models to be used in everything from mobile games to evaluating credit-checks.

Benefits of Deep Learning in the Cloud

Depending on the scale of your operations, implementing deep learning in the cloud can provide a number of benefits. This is particularly true for teams looking to adopt machine learning operations (MLOps) since pipelines and tooling are often already in the cloud. 

Increased scalability

One of the greatest benefits of using cloud resources for deep learning is the scalability that is possible. On-premises deployments are limited by local hardware and scaling can take significant time. In the cloud, however, you can scale as needed, temporarily provisioning hardware for particularly compute heavy tasks and scaling down during other times.

Additionally, cloud resources can provide scalability for hybrid workloads by providing burst capabilities as needed. This enables organizations to extend the value of their on-premises resources while still granting access to more performance.

Provider support for tooling

All major cloud providers offer some level of built-in support for existing ML and deep learning tools, including TensorFlow and PyTorch. This enables teams to continue working with the tools they are familiar with without limitations created by OS or infrastructure. 

Additionally, some providers offer enhancements for these frameworks. For example, pre-crafted notebooks for faster deployment. These enhancements enable teams to leverage provider tooling or resources to make implementation processes more efficient. 

Reduced barrier to entry

Machine learning in general and deep learning in particular, can require significant expertise and resources to implement. Cloud providers can help lower these barriers by offering pre-built services for developing, training, testing models. Some providers even offer ML as a service, enabling teams without ML developers to leverage the technologies available. 

Additionally, cloud resources can provide an easier entry point for deep learning operations. With cloud resources, you can test out methods and processes before making significant investments in hardware or tooling. You can also start small, and low risk, with cloud resources and scale up to on-premises investments once you better understand your hardware needs. 

GPUs in the Cloud

As cloud providers increase their support and options for deep learning implementations, organisations are beginning to take notice. While there are specialised providers available, the big three are where many organisations, especially those just getting started, should look.

Azure

Azure provides several choices for GPU-based instances. All of these instances are designed for high computation tasks, including deep learning, simulations, and visualisations.

In Azure, you can choose from three instance series:

  • NC-series—optimised for compute and network-intensive workloads. These instances can support OpenCL and CUDA-based applications and simulations. GPUs available include the NVIDIA Tesla V100, the Intel Broadwell, and the Intel HaswellGPUs.
  • NV-series—optimised for visualisations, encoding, streaming and virtual desktop infrastructures (VDI). These instances support OpenGL and DirectX. GPUs available include the AMD Radeon Instinct MI25 and NVIDIA Tesla M60 GPUs. 
  • ND-series—optimised for deep learning training scenarios and inference. GPUs available include the NVIDIA Tesla P40, Intel Skylake, and Intel Broadwell GPUs. 

AWS

AWS provides four instance options, available in multiple sizes. These include EC2 P2, P3, G3, and G4 instances. With these instances, you can choose to access NVIDIA Tesla M60, T4 Tensor, K80, or V100 GPUs and can include up to 16 GPUs per instance.

With AWS, you also have the option of using Amazon Elastic Graphics. This service enables you to connect your EC2 instances to a variety of low-cost GPUs. You can attach GPUs to any instance that is compatible for greater workload flexibility. The Elastic Graphics service also provides up to 8GB of memory and supports OpenGL 4.3.

Google Cloud

Although Google Cloud doesn’t offer dedicated instances with GPUs, it does enable you to connect GPUs to existing instances. This works with standard instances and Google Kubernetes Engine (GKE) instances. It also enables you to deploy node pools including GPUs. Support is available for NVIDIA Tesla V100, P4, T4, K80, and P100 GPUs.

Another option in Google Cloud is access to TensorFlow processing units (TPUs). These units are made of multiple GPUs. TPUs are designed to quickly perform matrix multiplication and can provide performance similar to Tensor Core enabled Tesla V100 instances. Currently, PyTorch provides partial support for TPUs.

Conclusion

There are a number of benefits to using cloud-based GPUs. Perhaps the most popular advantage is the scalability of the cloud. Instead of being limited to local hardware, you can quickly scale up or down without incurring on-prem overhead. You can also leverage cloud vendor support, and integrate with popular frameworks such as PyTorch and TensorFlow. 

Another popular benefit of cloud is that many vendors offer resources that can significantly save time. For example, you can use cloud AutoML tools to speed up some of your processes, and test out methods without investing too much time and costs. In this case, you also reduce risk by testing out your hypothesis. In short, cloud GPUs enable you to gain a higher level of scalability, save time, and avoid on-prem overhead.

Author Bio: Farhan Munir

With over 12 years of experience in the technical domain, I have witnessed the evolution of many web technologies, as well as the rise of the digital economy. I consider myself a life-long learner, and I love experimenting with new technologies. I embrace challenges with enthusiasm and outside-of-the-box mindset. I feel it is important to share your experiences with the rest of the world – in order to pass on the knowledge or let other folks learn from your mistakes or successes. In my spare time, I like to travel and photograph the world. YouTube

Are you using cloud GPUs in your development? Take our survey and share your experiences.

Categories
Tips

AR/VR Trends in the Ecosystem – Part Two

We continue to look into some of the AR/VR trends in the ecosystem, focusing on the main differences between developers and non-developers active in this space. You can view our first part of this article here.

22% of AR/VR non-developers are learning to code

15% of people involved in AR & VR as non-developers have zero knowledge of how to code, whereas 17% are actively coding to get things done. There is a large range of skills, but the biggest group here are those that are actively trying to build on their coding skills, with 22% of AR/ VR non-developers doing so. This indicates that no-code tools, whilst useful for getting things done, don’t cover the needs of more than 1 in 5 AR & VR practitioners who are subsequently learning to code to overcome these limitations. Despite the large amount of effort – and marketing – that has gone into positioning no-code tools as a solution for non-coders to get into AR & VR without worrying about writing code, a sizeable proportion of those who do get involved subsequently decide they need coding skills to realise their vision after all. This represents an opportunity for platforms aspiring to appeal to non-coders to create more functionality in these tools in order to capitalise on this under-served audience.

Amongst non-developers trying to improve their coding skills the most popular languages are Java (28%), C++ (26%), C# (25%) and JavaScript (22%). 17% don’t write code for their AR/VR projects, and 15% use a visual development tool. This indicates that non-developer AR & VR practitioners see the value in knowing how to code, but that they still want to create things while they’re learning.


AR/VR trends. Non-developers are actively learning Java, C++ and C#

Games is the most popular app category for developers and non-developers alike

AR & VR practitioners are primarily focused on creating entertainment and services products, but the primary focus is different for developers and non-developers. 77% of AR/VR developers are building products in the services category (such as business logistics products) whereas only 67% of non-developers are doing so. 

The most popular category for AR and VR practitioners is games & toys, with 52% of developers and 44% of non-developers working on products in this category. The picture is somewhat different for other entertainment products (such as moves and animation), with 65% of non-developers working on apps in this category and only 47% of developers operating here. This indicates that developers are using their coding experience to experiment and create games in AR and VR, whilst non-developers are inspired by other use cases.

Industrial applications (such as manufacturing and construction) for AR/VR are much less popular for both groups, but a larger proportion of developers are creating products in this category than non-developers. As AR and VR mature and stabilise, commercial applications will become more viable and we will see further innovations in industrial areas from developers and non-developers alike, but the pull of building entertainment apps will still be strong.


AR/VR trends. AR & VR practitioners mainly create entertainment & services products

Which AR/VR skills will you need to sharpen in 2021? Which tools do you think will be irrelevant as early next year? Our State of AR/VR survey is live. Spend 10 minutes sharing your experiences, we’ll donate $0.10 to Techfugees to for every completed response.

Categories
Tips

The Definitive Guide to Clean Code

Clean code is vital in every aspect of software development as well as mobile app development. A code that’s clean means it’s easy to understand and to change. Writing clean code is paramount in creating a successful and maintainable product. What is clean is, of course, subject to every developer’s take. Here is the definitive guide to clean code.

There are many considerations when it comes to what constitutes a clean code as well as the best coding practices. Nonetheless, there’s no particular distinction. Again, it is subjective to what a developer considers clean, based on the product being built.

Why Clean Code?

Clean code is important to care since code almost is never written only once and forgotten. Most of the time, you and others have to work on the code. Understanding of the code leads to work efficiency. The code is not only intended for computers but humans as well. Writing clean code means minimizing maintenance expenses. 

Furthermore, it means making it much easier when fixing bugs as well as working on code for many years a lot more enjoyable. In essence, clean code means making life easier for those involved in the project. However, it doesn’t mean being obsessed with clean code. It has to provide value, thus do not spend countless to make it perfect. 

Anyway, clean code doesn’t occur on the first try. The key is adopting a mindset that always strives to enhance the code that you work on. Clean code is paramount since it makes or breaks a project. 

Guide to Clean Code Characteristics

1. It should be elegant. It has to be pleasing to read, making a reader smile in such a way that a well-crafted music box would or even a well-designed car. 

2. It is focused. Every module, class, and function exposes a one-minded attitude that’s free from distraction and pollution by surrounding details.

3. It’s taken care of. Someone took the time to keep it orderly and simple. Moreover, they have paid close attention to details, meaning that they cared. 

4. It doesn’t have duplication.

5. Runs all tests.

6. Minimize entities like methods, classes, functions, and others. 

Writing Clean Code

Clean code follows and embraces specific practices. The practices are what make code readable, more comprehensible, cleaner, and simple. Implementing some of the following could bring positive results. 

  • Make readable code. Never neglect its comprehensibility and readability.
  • Utilize meaningful names for variables, methods, and functions 
  • Review the code regularly.
  • Allow one method or function to perform just one task. 
  • Use comments for clarification.
  • Be consistent.

1. Make readable code. It’s always a possibility that another would get your code or else if you’re working in the same software development company then you can re-use the code. It is of the best interest of all to write code in such a way that makes it easy to read as well as understand. Using whitespace is the easiest way. It isn’t a must to write code that looks minified. Rather, use line breaks, empty lines, and indentation to make the code structure more readable.

2. Utilize meaningful names for variables, methods, and functions. Meaningful means names that are descriptive enough that others would be able to comprehend the purpose of the function, method, or variable. In simple terms, the name itself should suggest the method, function, or variable is used or what it carries. Nevertheless, descriptive names do not mean you’re free to use as many characters as you wish. Rule of thumb would be limiting names to three or four words.

3. Review the code regularly. Simply writing code is not enough. The next step would be to maintain its cleanliness. Review it regularly, clean it, and try improving it. Otherwise, it would become outdated. The same as with the device, to keep them in the best shape requires regular updates.

4. Allow one method or function to perform just one task. Let every method or function do just one task. This may be simple but could change everything and help you write clean code, or at least something cleaner than before. Moreover, functionalities and methods also become predictable.

5. Use comments for clarification. Regardless of how you struggle to come up with meaningful methods, variables, or functions, on its own, the code still would not be clean and understandable as it should be. There would be lines that require elaboration. Using comments via the code helps to fix this. Comments explain to other people why you are writing them and the reason why it’s written in that manner. This way, people would not have to make a guess. Comments should be used only when necessary and not for explaining bad code. Writing endless comments won’t help transform a code that’s poorly written into clean code. A bad code should be fixed by improving it, not adding instructions on how to use it. 

6. Be consistent. Stick to specific coding styles or practices and use it everywhere. It’s not a good idea to use different styles or practices in another project. It’s not as smooth and natural as it could be getting back to old code. Select coding practices set and stick to them in all projects. It would be much easier to return to an older code and proceed where you left off or to improve code. It’s a good idea to experiment on new coding practices. Furthermore, it could help look for better ways to do work. Nonetheless, it would be preferable to experiment with different practices on a separate project or exercise instead of on the main project.

Why Strive for Clean Code

Clean code is easy to understand and readable by everyone. Writing clean code is a critical mindset and it takes practice. Nobody is perfect, including you. There is always an opportunity to refactor or improve code when you review it after a few days or a few weeks. Therefore, begin writing a clean code as you can from the first line to work more on improving performance and logic. 

Pros

1. Better use of time. The first recipient of clean code is, of course, the programmer. Working on a project for months sometimes makes it easy to forget what you’ve done with the code. This is particularly true when a client comes back with changes. Making changes is easier with clean code lines. 

2. Easy debugging. Bugs are inevitable whether you write clean or dirty code. Clean code, however, makes for faster debugging, whatever your expertise or experience may be. Managers or colleagues could help and jump in to help you. 

3. Easier for new members of the team to get on board. Principles of clean code help to get onboard a new programmer. It helps to get a new programmer on board with clean code. No documentation is required to understand the code and a newbie could jump directly into it. This saves time on training and for a new member to adjust to the project. 

4. Much efficient maintenance. Maintenance doesn’t mean fixing bugs. Any project would need changes, new features to existing ones. In any software project, maintenance is the major cost. New features or additional ones always are an afterthought. Clean code makes fast and easy maintenance. 

5. Feel good. Clean code makes you feel good and confident to share it with others or with a client. You need not fear of breakdown and you would also be able to fix defects faster and this means you get to enjoy programming more. 

6. Easier to solve problems. The approach to problem-solving changes when writing clean code. Furthermore, software design and algorithm design become more intentional and elegant.

7. Communicated ideas. A clean code minimizes the change of misunderstandings when working with other developers and programmers. Furthermore, this also means lesser bugs to deal with in the future. 

Going Further with Clean Code

Majority of the basics in making clean code are common sense. Nonetheless, even with the most basic software, success requires paying attention and focusing on details. Products increasingly have more at stake than pride and money. In today’s age where we have autonomous vehicles on the road, the rules on code structuring and writing have gained relevance. Furthermore, more diligence should be applied to security-critical and safety systems. 

Industries like defense, automotive, aerospace, banking and finance, and medical device manufacture have stringent coding compliance standards. The code should meet industry standards. Source code could be scanned using static code analysis tools. Furthermore, the tools could also detect vulnerabilities in security, hidden defects, and ensure adherence to the best coding standards. 

Undoubtedly,  custom software development companies create products for their clients. Clients could do whatever they want with the software or product. They could use it internally or even sell it to somebody else. Most importantly however is that they could modify or extend it themselves. 

To be able to do this, they need a clean code. Thus, when building software, it is of utmost importance to deliver clean code at all times. Again, clean code truly matters. A clean and readable code enables continuity of the product. Furthermore, it makes it possible for another person to continue the work and enjoy doing so. The result is a lesser bug and easier to maintain software. 

Did you enjoy this guide to clean code? Is there anything else that a developer should consider?

About

Working as a manager in a software development company Tatvasoft.com. Vikash always has a passion to explore more about software  development and likes to share his bylines among relevant audiences. He is a full time blogger and you will find his informative bylines around many corners of blogs related to development. You can always find him on Twitter


What will be the biggest dev trends in 2021? Help us find out! Take the new Developer Economics survey, answer a few questions about dev tools, platforms, technologies for a chance to win cool prizes!

Check out our latest research reports and graphs, based on data from developers who took our global surveys.

Categories
Tips

Infographic: What did developers do during Covid 19 outbreak?

Evers since the Covid-19 outbreak took on global pandemic proportions, a lot has changed about how developers work, communicate, and stay sane, above all. There are certain stereotypes about developers (i.e developers are “loners”, most working remotely, they get most of their entertainment online anyway etc) that make it all too easy to assume that quarantines were easy on developers. Did they feel the impact of lockdowns and social distancing as sharply as other people? We wanted to debunk these stereotypes and ask developers what really affected their lives during these uncertain times.

To understand what’s happening, we ran a small poll in May 2020 asking the members of our community to describe their day-to-day during quarantine. How did the new social distancing rules affect their work, education, entertainment, and lifestyle choices? This mini-survey was intended to be fun and entertaining first and foremost. 138 developers from our community took part. We thank them once again for being good sports and helping us create these light-hearted insights on an otherwise very heavy subject.

The results are summed up in this Covid-19 infographic, which we hope you’ll find amusing.

You may be nodding your head as you scroll down and think “yes, me too!” or you may have a very different experience that you’d like to share. Whatever the case, we invite all developers to take part in our new Developer Economics survey, where we investigate new trends in software development and what the future holds.

covid19 infogrpahic, covid19 quarantine, covid19 poll, covid19 survey, software developers, developer survey, developer poll

Do you like this infographic? Feel free to share it on social media and don’t forget to tag us as the source!

Categories
Tips

Professional Front-end Developer Guide to the development triangle

If you’re interested in dabbing your toes in the front-end development, this post if for you. Our community is rich with developers covering all major areas of development, so we reached out to one of our loyal members to share a few front-line tips with you via this mini professional front-end developer guide!

Dominic Myers is a professional front-end developer with a wealth of experience. He spends most of his day being fascinated by, developing and implementing the newest trends in web development, primarily HTML, CSS and JavaScript, or whatever will get the job done. Here he shares his professional front-end developer guide to the front-end development triangle.

A Professional Front-end Developer Guide

I’ve spent the past couple of years writing, editing (and editing some more) a book about front-end development (which you can purchase here or here), which is finally making it’s slow and ponderous way into the light. That use of words suggests that it’s a vast and weighty tome of a book – and it certainly was when I first submitted it – but after all that editing, it’s finally down to a fighting weight, and I think it’s a better book for that. One thing I was keen to point out is the triangle of technologies we, as front-end developers, need to be conversant. In much the same way that fire isn’t possible without fuel, heat and oxygen, front-end development is impossible without a more than cursory knowledge of HTML, CSS and JavaScript (JS). 

HTML

In the dim and distant past, when attending the introductory lecture for a conversion Computer Science MSc, I didn’t understand why the rest of the students sniggered when someone answered HTML when asked if we had any experience of programming. Once the chuckling had subsided sufficiently, the lecturer was keen to point out that while HTML was not a programming language, with its lack of all those things recognisable as being associated with programming languages such as variables and functions it couldn’t be; it was still a language – be it a markup language. As such, it involves a certain level of abstraction. 

The writer of HTML needs to have an appreciation of the milieu in which it will reside: the browser. Browsers themselves are unusual beasts with all sorts of wild and weird characteristics. The same browser might act differently on different architectures, or in identical hardware but with varying levels of user privileges, or with plugins installed. The unfortunate HTML author might even have to deal with superseded versions of browsers (Yes, IE11, I’m looking at you! I spend a significant amount of time massaging ES6 into ES6 for IE11, most often by hand but sometimes with Babel – but even Babel has its limits).

Along with the vagaries of the browser, an appreciation of what the browser will do with HTML is necessary. It isn’t merely rendered line by line to the user interface; instead, it is converted to Document Object Model (DOM). I like to think of the DOM as something like a tree with a trunk quickly splitting into two – one being something of a stub and the other adorned with a whole plethora of branches, sub-branches and twigs. This second branch is when the HTML writer gets to play, and the palette of elements they get to play with grows larger all the time! I guess my analogy might get strained though, my knowledge of Dendrology is limited, but suffice it to say that there are many HTML elements, some of which have fallen from favour, while still others have found favour and been embraced.

Without this tree as a foundation, front-end development is nigh on impossible. I can’t tell you how many times I’ve seen CSS or JS courses suggesting that a rudimentary knowledge of HTML is a requirement. Rudimentary knowledge? HTML is our bread and butter and, while we can use CSS to get a `div` to behave like a `span`, why not use a `span` in the first place? But this brings us neatly to our next side of the triangle: CSS.

CSS

It is arguable that if my fellow student had said CSS instead of HTML, then the laughter might’ve been even louder – this was a few years ago before you say anything. CSS acts as adornment for our DOM tree; it gives its elements’ colour if you will. That’s not to say that altering the colouring is its limit – but it’s still essential. Also, please take note of when they were laughing; CSS was then not Turing Complete (footnote: Turing Completeness denotes a language where a question is asked and answered – there is no guarantee how long that answer might take though). CSS3 was demonstrably Turing Complete in 2011, though follow through on the links please, its a fascinating read.

CSS3 is profoundly compelling and allows the canny front-end developer to avoid all sorts of weird and wonderful JS trickery. Trees in autumn can be bare and stark but can still hold a beauty all their own, but trees in the flush of Spring, Summer and Autumn are nigh on always a delight to behold. CSS serves as the foliage and, while I appreciate aesthetics plain HTML I love decorated trees, all the while knowing that without the foundation of HTML – the trunk and branches if you will – all that gorgeous adornment would be on the ground, blowing in the wind. 

via GIPHY

CSS has taken over from JS in some areas, though there is still a tendency for front-end developers to reach for JS rather than utilise CSS.

I can understand this: if all you’ve got is a hammer, everything looks like a nail. If I can do something quick and dirty using JS rather than research the CSS technique, then I used to do that. I did before I clocked just how performant it was to use the native capabilities inherent in CSS, it’s lightning-fast so learn and use it, please!

That brings us nicely to the next side of our triangle: JS.

JS

I dread to imagine what the reaction would’ve been to my fellow student saying JavaScript. I dare say snorts of derision rather than any form of laughter. JS has a history of being seen as something of a joke language, which is a shame as it is rather elegant – especially now.

If HTML is the bare tree and CSS is the foliage, then JS allows us to act like tree surgeons with superpowers! With JS we can make all sorts of alterations to the DOM (the DOM is, after all, merely an API for HTML). We can add, subtract and alter the constituents of the HTML with JS, we have superpowers with JS. But take heed of my caution above regarding CSS, just because you can do something with JS, first please check you can’t do the same thing with CSS! After all, with great power comes great responsibility! 

via GIPHY

There is an excess of books and courses on JS and the frameworks and libraries that have sprung up about it. Strangely, such an overabundance can present something of a barrier to those seeking to enter front-end development. With a whole feast presented to you, it’s easy to find an old and obsolete resource and end up eating something well past its sell-by date, getting food poisoning and resolving to give it up as a bad job.

But, should you find something bright, shiny and new and that works off the bat, or drink the kool-aid by some other route, the sheer power of JS can mean that you leave good old HTML and CSS by the roadside. You might even get tempted to start implementing some CSS-in-JS and never have to touch a CSS file throughout your career, only looking at the MDN Web Docs to find out what the JS name for a CSS property is.

Thoughts

I think it is this disconnect between JS and the other two sides of the triangle that prompted Chris Coyier to write about “The title ‘Front-End Developer’ is obsolete.”. Something of a provocative title and one that he doesn’t endorse but uses as a launchpad to explore the confusion regarding the term “front-end developer”. Most, if not all, job postings for front-end developers will ask for one of three things: Angular, React or Vue. That list is likely to change though, and it’ll change at a rate of knots, in much the same way as the things themselves and their associated toolchains will also change. Sometimes the postings won’t even mention HTML or CSS or will suggest candidates should know SCSS, further alienating those whose bread and butter is HTML and CSS.

Does this mean that we’re looking at a further sub-division within development? The need for developers dedicated to working with the front-end led to a separation of developers into front- and back-ends. Are we now seeing a further split between HTML and CSS developers and JS developers, with HTML and CSS developers getting short shift? I’m not so sure TBH, those dedicated JS developers do generate HTML and CSS, but do so via JS, leading to a further question of how well they appreciate the nuances of both HTML and CSS? Perhaps more of a focus during preparation for the career on these would suffice. HTML is rock solid, and CSS standardised nowadays so the necessity of knowing how anything other than evergreen browsers interprets identical HTML and CSS.

I’m not sure about what to do to address the situation of this further fracturing, or even if it’s worth discussing – I’m just conscious of it, and I’m wary of losing something. JS gives you superpowers and enables the canny developer to spin a whole DOM out of nothing, made up of HTML and CSS. The superpowered JS has some mean powers but still, in the end, relies upon a thorough knowledge of both HTML and CSS, it isn’t a one-man-band. All three together and employed skillfully are an orchestra with a profound impact and rich timbre. We owe it to ourselves as developers to embrace the full triangle. a cursory knowledge of HTML, CSS and JavaScript (JS). 

About

Dominic Myers is a front-end developer with a wealth of professional experience. He spends most of his day being fascinated by, developing and implementing the newest trends in web development, primarily HTML, CSS and JavaScript, or whatever will get the job done. He shares his knowledge online via his blog and forums such as Stack Overflow.

If you are involved in front-end and want to share your views, visit our latest survey and help shape the trends.

Did you enjoy this professional front-end developer guide? Do you have a topic that you are passionate about and would like to share with our developer audience in our blog? Get in touch!

Categories
Tips

AR/VR Trends in the Ecosystem – Part One

Augmented Reality (AR) and Virtual Reality (VR) have captured the public imagination for decades; from the Holodeck in Star Trek to Ironman’s Heads Up Display, this technology is synonymous with visions of the future. Recently however, AR & VR processing has become commonplace on smartphones and companies like Oculus & Sony have released consumer-quality headsets. In this post we take a look at some of the AR/VR trends in the ecosystem, focusing on the main differences between developers and non-developers active in this space. Let’s look at those AR/VR trends.

AR & VR are the smallest individual software sectors

Considered individually, Augmented Reality and Virtual Reality remain the smallest software sectors out of the ones we research (the others being mobile, desktop, web, games, backend, industrial IoT, consumer electronics, data science & machine learning). Even when combined, AR and VR (AR/VR) are only marginally bigger than Consumer Electronics, the next smallest sector. Only 0.4% of people are involved solely in AR or VR, the rest are involved with at least one other development area.

AR/VR trends – Only 0.4% of developers are involved solely in AR and/or VR

Of the 9% of people involved with AR or VR almost half (46%) are involved with both AR and VR. This shows that there is a significant overlap in the skills needed to work in these sectors. There are more people involved solely with VR (31%) than with AR (24%). AR is slightly less mature than VR and there are some technical challenges in AR (occlusion, optics & object registration, for example) which are still being resolved, this also means that there is a smaller market for AR products, as the technology is less established. This results in a slightly higher barrier to entry and subsequently a smaller number of people involved in AR than for VR only.


AR/VR Trends - 46% of AR/VR practitioners are involved with both AR and VR

AR & VR trends – practitioners are mostly hobbyists

One of the defining features of AR & VR practitioners is their diverse involvement in different development areas. As previously discussed, the number of people involved solely in AR & VR is very small, but in fact, many of them are also involved in multiple development areas. Over 60% of practitioners involved in AR and VR are involved in 5 or more sectors in total. This is a large contrast with respondents who don’t work in AR or VR, where only 9% only are involved in 5 or more sectors.

Most people who are involved in AR or VR are hobbyists, and not just in AR & VR. These people are more likely to be hobbyists in every other sector than people not involved in AR or VR. They are technology enthusiasts who like to experiment outside of their professional duties, and are currently experimenting with AR & VR, potentially with a view to incorporating AR & VR into their existing development projects.

Looking at this from the other side, 28% of VR professionals also consider themselves to be hobbyists in the same sector. Out of AR professionals, 24% take on AR projects in their spare time as a hobby. This is higher than most other sectors, with machine learning being the next highest at 26%, then games at 25%. This shows that AR & VR practitioners are enthusiastic about the sector, often having passion projects on the side.

57% of AR & VR hobbyists work professionally in at least one other development area

We also see more diversity in the type of roles that AR & VR practitioners do. Because AR & VR sit at the intersection of arts and technology, practitioners often fulfill hybrid (both technical and non-technical) roles. In fact 35% of AR practitioners fulfill a hybrid role. Subsequently, people involved in AR/VR are less likely to be ‘Pure Developers’ (people solely fulfilling developer-type roles) than those involved in other sectors. This difference is especially pronounced for respondents working in VR or in AR and VR, with only 34% and 38% respectively working solely in developer roles, compared with 50% of respondents working in AR only.


AR/VR trends - practitioners often fulfill hybrid roles

Practitioners who are involved in VR only, or VR and AR are more than twice as likely as their counterparts who are only involved in AR to be in non-developer roles. This shows that non-developers tend to favour working in VR in some capacity.

Drilling down into the roles, we see that 49% of AR practitioners work as programmers or software engineers, compared with only 37% and 32% respectively for VR practitioners and those who work in both AR and VR. Many AR practitioners are also involved in web & mobile development and machine learning. This suggests that these coders are interested in AR from a technical point of view, looking to challenge themselves by using the latest technology or to implement AR in their projects.

On the other hand, VR practitioners and those involved in both AR and VR are more than twice as likely as AR practitioners to be game designers or work as product managers. The popularity of these roles reflects the quick uptake of VR by the game market – moving from being an emerging technology to generating revenue.

Not only do AR & VR practitioners hold different roles compared to people involved in other sectors, but they also wear a lot of different hats. More than 20% of people involved in AR or VR take on 4 or more roles, compared with only 12% of people involved in other sectors. We already know that AR & VR developers are often passionate hobbyists, but it’s also clear that they have diverse interests and skills. This diversity comes from the fact that as AR and VR development technology matures, tools are appearing which require fewer technical skills to create an AR or VR product. This attracts non-developers who can more easily realise their vision.

AR & VR practitioners take on more roles than people working in other sectors

What other AR/VR trends are there?

Almost as many AR & VR developers use 3D animation software as use IDEs

While there is some overlap in the technologies used by developers and non-developers involved in VR, none of them have a strong appeal for both audiences. The Oculus technology suite comes closest to being the go-to platform for both developers and non-developers, with over 35% of each audience using the platform. Playstation VR, Windows 10 Mixed Reality & Google Daydream all attract a good proportion of non-developers (36%, 28% and 26% respectively), but fail to appeal to VR developers. This landscape creates an opportunity for a technology vendor willing to invest in widening support and access for one (or both!) audiences, as a unified tech stack would provide large efficiency benefits to integrated teams by integrating with other tools and platforms, streamlining training needs and reducing the variety of tools being used.


AR/VR Trends - Oculus leads across all practitioners, but PlayStation VR is equally popular among non-developers

Unity Mobile AR, AR Core and AR Kit lead the pack of software tools for people creating AR products, but all of these tools are favoured more by developers than by non-developers. This suggests that there is space in the AR software market for a tool which allows non-developers to more easily realise their creative vision.


AR software tools appeal more strongly to AR developers than non-developers

Over half of developers use game engines and 48% use 3D modelling and rendering software. The high uptake of these technologies amongst AR/VR developers is testament to the powerful efficiency gains available from the abstraction they offer, as well as that AR, and especially VR, lend themselves to game development.

Over 50% of AR & VR developers use game engines

We’ve already seen that practitioners that undertake developer and non-developer roles (hybrid developers) make up a sizeable proportion of those involved with AR & VR, and this is validated by the popularity of 3D animation software (39%) and designer tools (30%) amongst the technologies used by AR & VR developers. In fact, almost as many AR & VR developers here use 3D animation software as use IDEs!

Backend-as-a-service, ML APIs and app store analytics are all used by less than 15% of AR & VR developers. The usage rate of app store analytics for AR & VR developers is 3 percentage points lower than for game developers, and 10 percentage points lower than for mobile developers. This suggests that AR/VR developers are focusing on getting the basics right, rather than trying to extract maximum value from their apps’ marketing funnel.


Graph depicting technologies used by devs and non devs

We see some overlap in the tools used by non-developers; 49% use 3D modelling and rendering software, 43% use game engines and 42% use 3D animation software. The high usage rates of the more artistic technologies is to be expected, given that these people are, by definition, not developers.

The Adobe toolset is the most popular software tool amongst non-developers, but the next three most popular software tools are all SDKs used by 24% of AR non-developers (ARCore, ARkit and Unity Mobile AR). This begs the question, do non-developers involved in AR & VR know how to code?

What are your biggest pain points in getting into AR/VR development? You can share your experiences in our AR/VR survey.

You can read more AR/VR trends in our State of the Developer Nation report.

Categories
Tips

Top Companies Contributing to Open Source

Who are the top companies contributing to open source? This blog post looks at how CodersRank used publicly shared data to answer this question, and how they created a series of data visualization videos.

The boom of open source software brought a change in technology that shaped the world as we know it today.

Open source exists thanks to the hard work of dedicated programmers and developers. It has become the foundation of cloud computing, software-as-service, next generation databases, mobile devices, the consumer internet, and more. 

undraw open source
Source: undraw.co

We, at CodersRank, are great admirers of open source. Almost every one of us contributes to open source regularly, and we sometimes work on a project or two together. 

In this blog post, we’ll introduce you to a video that we created. This video gives you a visual representation of companies that contributed the most to open source since 2012. If you find this data interesting, then you’d probably love to know the methodology behind it. We’ll show you exactly how we gathered the data and then how we gave it a visual spin. 

Video: Top Companies Contributing to Open Source

The video, “Top Companies Contributing to Open Source | 2012-2019”, is part of a series of data visualization videos that we came up with at the end of 2019. We made these out of curiosity, after realizing that you could see certain trends forming if you put together some of the publicly shared data. 

Haven’t seen the video? Here it is:

Behind-the-scenes of our method

This will be a quick overview of our method – please see the actual code used further down this page.

Measuring contributions

In measuring the contributions we only considered the commits. We know that there are many other ways to contribute to a project not just commits but in this particular case we wanted to focus on the commits.

Defining contributing authors

We relied on the email addresses of the authors. The second part of the email is usually the company’s domain.

Assigning commits to a company

There are around 2.4B public commits in GitHub (since 2011) and we had to analyze each and every one of them to answer this question. Thankfully, not manually!

Using the GitHub API to extract that amount of data would have been impossible. Thanks to the GitHub Archive Project, all the public GitHub events are stored in a publicly available BigQuery database. Using SQL to extract data made the process easy and painless.

Cleaning up the data

After we counted the commits for each company, the data needed to be cleaned. First, we excluded email providers like gmail, hotmail, yandex etc. Then, we excluded a few more, as there were some cases where the commits were made by bots.

Converting the results to the expected format

We used Flourish to create the videos. The data had to be converted into a format that is acceptable for Flourish (i.e.: handling months with no data from a given company).

Implementation

Step 1: get the commits/domains

Top companies contributing to open source - PushEvent in BigQuery-  get the commits/domains

The payload column is what we needed here, since it contained the email address. In our example it is a454492e42fd9810e577ebee548c7e59bd883bca@live.com.au. GitHub hashed the first part of the email, but we didn’t need that anyway, because we were only curious about the domain-level information. 

The query to count the commits/domain name looked like this:

## pre-2015 API
CREATE TEMP FUNCTION
 json2array(json STRING)
 RETURNS ARRAY<STRING>
 LANGUAGE js AS """
         return JSON.parse(json).map(x=>JSON.stringify(x));
       """;
WITH
 export_domains AS(
 SELECT
   DATE_TRUNC(DATE(created_at), month) AS month,
   emails,
   ARRAY(
   SELECT
     REGEXP_EXTRACT(x, "@(.*)")
   FROM
     UNNEST(emails) x
   WHERE
     REGEXP_EXTRACT(x, "@(.*)") IS NOT NULL) AS domains
 FROM (
   SELECT
     * EXCEPT(array_commits),
     ARRAY(
     SELECT
       JSON_EXTRACT_SCALAR(x,
         '$[1]')
     FROM
       UNNEST(array_commits) x) emails
   FROM (
     SELECT
       created_at,
       json2array(JSON_EXTRACT(payload,
           '$.shas')) array_commits
     FROM
       `githubarchive.day.20130101`
     WHERE
       type='PushEvent' )))
SELECT
 month,
 flattened_domains AS email_domain,
 COUNT(flattened_domains) AS domain_count
FROM (
 SELECT
   month,
   flattened_domains
 FROM
   export_domains
 CROSS JOIN
   UNNEST(export_domains.domains) AS flattened_domains )
GROUP BY
 month,
 email_domain
ORDER BY
 month,
 domain_count DESC

After 2015, the format of the payload changed a bit and required a slightly different query:

## post-2015 API
CREATE TEMP FUNCTION
 json2array(json STRING)
 RETURNS ARRAY<STRING>
 LANGUAGE js AS """
         return JSON.parse(json).map(x=>JSON.stringify(x));
       """;
WITH
 export_domains AS(
 SELECT
   DATE_TRUNC(DATE(created_at), month) AS month,
   emails,
   ARRAY(
   SELECT
     REGEXP_EXTRACT(x, "@(.*)")
   FROM
     UNNEST(emails) x
   WHERE
     REGEXP_EXTRACT(x, "@(.*)") IS NOT NULL) AS domains
 FROM (
   SELECT
     * EXCEPT(array_commits),
     ARRAY(
     SELECT
       JSON_EXTRACT_SCALAR(x,
         '$.author.email')
     FROM
       UNNEST(array_commits) x) emails
   FROM (
     SELECT
       created_at,
       json2array(JSON_EXTRACT(payload,
           '$.commits')) array_commits
     FROM
       `githubarchive.day.20150102`
     WHERE
       type='PushEvent' )))
SELECT
 month,
 flattened_domains AS email_domain,
 COUNT(flattened_domains) AS domain_count
FROM (
 SELECT
   month,
   flattened_domains
 FROM
   export_domains
 CROSS JOIN
   UNNEST(export_domains.domains) AS flattened_domains )
GROUP BY
 month,
 email_domain
ORDER BY
 month,
 domain_count DESC

The result looked like this:


Row
month email_domain domain_count
1 2015-01-01 gmail.com 131357
2 2015-01-01 users.noreply.github.com 8802
3 2015-01-01 python.org5786
4 2015-01-01 hotmail.com4942
5 2015-01-01 fhda.edu 3888
6 2015-01-01 yahoo.com3216
7 2015-01-01 etudes.org 2736
8 2015-01-01 qq.com 1955
9 2015-01-01 sly.mn 1908
10 2015-01-01 foothill.edu 1848

Step 2: exclude email providers

The heavy lifting was done by BigQuery. We exported the results into a .csv file and used the good old Jupyter Notebooks to clean up the data.

As you can see in the example result, not surprisingly, the first one was gmail.com. Our next task was to remove the email providers from the list. 

We used a GitHub contribution of the most popular email domains for the cleanup: https://gist.github.com/tbrianjones/5992856/

And we also added some other blacklisted domains (excluded_domains.txt):

.(none)

91177308-0d34-0410-b5e6-96231b3b80d8

samo-laptop.(none)

dd0e9695-b195-4be7-bd10-2dea1a65a6b6

ubuntu.(none)

b8fc166d-592f-0410-95f2-cb63ce0dd405

b9a71923-0436-4b27-9f14-aed3839534dd

b2dd03c8-39d4-4d8f-98ff-823fe69b080e

0b4bb1d4-4e5a-0410-9cc4-b2b747904278

709f56b5-9817-0410-a4d7-c38de5d9e867

iki.fi

Gmail.com

none

example.com

1a063a9b-81f0-0310-95a4-ce76da25c4cd

localhost.localdomain

localhost

localhost.(none)

home

b8457f37-d9ea-0310-8a92-e5e31aec5664

li7-202.members.linode.com

g

users.noreply.github.com

us.door43.org

mailinator.com

smullindesign.com

review.openstack.org

nyarlabo.com

boston.com

li.gugod.org

niob.xnis.de

sly.mn

kazer.org

recoil.org

tsaousis.gr

rituwall.com

cbrese.com

renovateapp.com

scrapers.everypolitician.org

Step-by-step code walkthrough

Load BigQuery results

from tqdm.notebook import tqdm
 
import pandas as pd
import numpy as np
 
tqdm.pandas()
 
df = pd.read_csv("./email_domains_large.csv")

Merge the list of domains we want to exclude:

free_providers = list()
with open("./free_email_provider_domains.txt", "r") as f:
    for line in f.readlines():
        free_providers.append(line.strip())
excluded_emails = list()
with open("./excluded_domains.txt", "r") as fe:
    for line in fe.readlines():
        excluded_emails.append(line.strip())
free_providers = free_providers + excluded_emails

Add a new column to the dataset, whether the domain is a free email providers’ domain

df["free"] = df["email_domain"].progress_apply(lambda x: x in free_providers)

Create a list without the email providers

df_filtered = df[(~df["free"])].copy()

Add a row counter and limit the data to those domains that appear at least once among the top 30. This will make the final dataframe smaller and easier to handle.

rn = list()
for _, df_tmp in df_filtered.groupby("month"):
    t = list(range(df_tmp.shape[0]))
    rn += t
df_filtered["rn"] = rn
domains = np.unique(df_filtered[df_filtered["rn"] <= 30]["email_domain"])

Step #3: format data

As it was mentioned before, we used Flourish to create the video. In some cases there were empty months (the company didn’t have any commits) and Flourish expected the columns to be months not companies. So we had to make this transformation too.

Step-by-step code walkthrough

df_final = df_filtered[df_filtered["email_domain"].apply(lambda x: x in domains)].copy()
df_final["month"] = pd.to_datetime(df_final["month"])
date_range = pd.date_range(np.min(df_final["month"]), np.max(df_final["month"]), freq="MS")
 
temp_df_list = list()
for _, repo_data in df_final.groupby("email_domain"):
 
    df_temp = pd.DataFrame()
    df_temp["month"] = date_range
    df_temp = df_temp.merge(repo_data, on="month", how="left")
    df_temp.fillna(method="ffill", inplace=True)
    df_temp["email_domain"].fillna(method="bfill", inplace=True)
    df_temp.fillna(0, inplace=True)
    temp_df_list.append(df_temp)
 
df_full_data = pd.concat(temp_df_list, ignore_index=True).sort_values(["month", "domain_count_rolling"], ascending=[True, False])

df_chart_race_final = pd.DataFrame()
df_chart_race_final["email_domain"] = list(df_full_data["email_domain"].unique())
 
for current_month, monthly_data in df_full_data.groupby("month"):
    month_name = current_month.strftime("%Y-%m") 
    
    df_temp = df_full_data[["month", "email_domain", "domain_count_rolling"]].query("month == @current_month")
    df_chart_race_final = df_chart_race_final.merge(
                df_temp.drop("month", axis=1).rename(index=int, 
                                             columns={"domain_count_rolling": month_name}), 
                on="email_domain", how="left")

The full notebook can be found here.

Final word

More and more companies are recognizing the importance of open source software development and are committed to support it. 

We hope that visualizing just a slice of the data that these amazing men and women generate round the clock is a way to acknowledge their hard work. Thanks to them and the millions of hours they invested to build open source products, we get to use our everyday apps and software seamlessly. 

Thank a developer today!

About

Adrienn Tordai
Growth Marketer / Pizza Enthusiast @CodersRank. I love the Blue Jays, books, and The Office. Tell Elon I said hi. Always waiting for a Steam sale. | CodersRank: Our goal is supporting DEVELOPERS’ growth by their always up to date, professional CodersRank profile

Categories
Tools

DevOps CI/CD usage trends

To understand DevOps CI/CD usage trends, SlashData has, over the past three and a half years, tracked the usage of continuous integration & delivery (CI/CD) tools and services among mobile, desktop and web developers. While DevOps is technically a culture rather than a set of tools, CI/CD is at the core of the collaboration process between operations and developers. These tools enable some of the most important benefits of the shift to this new culture. As can be seen in the chart below, the majority of developers are not using these tools and usage is not growing. 

While many firms in various studies have indicated that they are adopting DevOps, our data suggests that this shift in culture is not ubiquitous across organisations. Has the use of CI/CD tools reached as many developers as it can or are there certain market barriers?

CI/CD Usage

% of developers using CI/CD tools

DevOps CI/CD usage trends - CI/CD adoption trends by mobile, web and desktop developers

Profile of CI/CD users

Understanding the profile of developers using CI/CD compared to those that are not can provide more insight into why usage is not as high as one might expect. In general, developers who use CI/CD tools are professionals working for companies with larger development teams and are more experienced compared to developers not using CI/CD.

Data from our most recent survey shows that developers that are using CI/CD tools are much more likely to be professional developers than those that are not. Web developers using CI/Cd tools are 20 percentage points more likely to be professional developers compared to developers not using these tools. For mobile and desktop developers this differential is 22 and 18 percentage points respectively.

DevOps CI/CD usage trends - Profile of CI/CD users DE
% of users and non-users of CI/DE who are professional developers

Our most recent survey also shows that 46% of developers using CI/CD tools work for organisations with more than 20 people involved in software development. This compares to only 32% of CI/CD non-users who work for firms with more than 20 developers. The fact that CI/CD users are less likely to work in smaller developer teams points to lower demand at small firms. This may be due to less complex development operations requiring less automation and integration of developer and IT teams.

DevOps CI/CD usage trends - Number of people involved in software development in company

58% of developers using CI/CD tools work for firms with 10 or more people involved in software development

How big are the development operations adopting CI/CD?

Developers who use CI/CD are also more experienced than developers who have not integrated CI/CD into their development process. Of web developers using CI/CD, 44% have six or more years of experience compared to just 28% of developers who are not using CI/CD. For desktop developers, we see the same trend, with 49% of these developers using CI/CD possessing six or more years of experience compared to 34% of developers who are not using CI/CD. Mobile developers using CI/CD are slightly less experienced, but the overall trend holds true, with 49% of developers using CI/CD having three or more years of experience, 17 percentage points more than those not using CI/CD.  

Experience in web development, desktop development and mobile development

As developer operations adopt DevOps culture, to improve the odds of success, developers should have an understanding of the entire development process. This may be easier for more experienced developers to manage. 

While some may struggle, other developers have fully embraced the shift to DevOps and bring a variety of skills to the table and have carved out a role as a DevOps specialist.

The DevOps Specialist

DevOps specialists play an important role in driving DevOps culture and are often evangelists. These practitioners are relatively uncommon with only 5% of developers in our survey identifying as having this job. This lack of evangelists and experts may be an important factor limiting the culture shift throughout an organisation.

Finding professionals with diverse skill sets to occupy these roles may be contributing to the low number of DevOps specialists. One of the keys to successful DevOps implementation is merging of cultures so it is important that professionals driving the process have an understanding of and empathy for how both IT and developers work. 

Developers who identify as a DevOps specialist are a diverse group and indicate that they play additional roles in organisations beyond just DevOps specialist. The majority of survey respondents identify as software developers but DevOps specialists are 8 percentage points more likely to also be a software developer compared to developers who do not identify as a DevOps specialist.

DevOps specialists are also much more likely to be architects, administrators (both data and system), engineers and testers. These skills play an important role in the implementation of DevOps. Architects are needed to automate processes, administrators are required to manage release schedules and testers and QA engineers are needed to test software as it moves through each step of the iterative development process.

What else do DevOps specialists do?

DevOps CI/CD usage trends - What else do DevOps specialists do? % of DevOps specialists in other roles vs developers that are not DevOps specialists

Digging deeper into the DevOps specialist’s skill sets we can also observe that many have more than one role beyond DevOps specialist. From the table below we can see how these developers describe themselves and how their roles overlap. Twenty-three per cent of DevOps specialists are both programmers and system administrators and 27% are programmers and architects. Having development skills and an understanding of how to manage and design systems are an important combination of skills for leading a DevOps strategy. 

DevOps specialists’ additional roles

DevOps specialists' additional roles

While DevOps is a very popular strategy already adopted by many organisations, based on the number of developers using CI/CD tools, not all developers are buying in. The DevOps culture has been slower in reaching less experienced developers and ones at smaller firms. The need for more experienced professionals with diverse skills may be a barrier to more developers benefiting from DevOps tools such as CI/CD. 

If you are interested in learning more, here is some additional reading.You can voice your opinion in our current survey to shape the next State of the Developer Nation report and contribute to future DevOps CI/CD usage trends.