Jina AI provides cloud-native neural search solutions powered by state-of-the-art AI technology. The company says its neural search framework called Jina empowers anyone to build SOTA and scalable neural search applications in minutes. In a conversation with Abbinaya Kuzhanthaivel, Pradeep Sharma, senior developer relations expert, Jina AI, touches upon his open source journey, the range of skills needed in this industry, the challenges in the open source world, how open source helps to speed up project development, and more.
Q. Can you tell us a bit about yourself and why you chose to work on open source projects?
A. I am working with Jina AI, the cloud native neural search solutions provider, as a developer relations expert for building relationships with the developers of our open source projects.
I graduated from IIT Roorkee in 2012 as an electronics and communications engineer, and always had a keen interest in programming. I joined a FinTech startup as a software engineer and built a lot of projects there. When one of my projects worked really well, I built my own startup and worked in it for about two years. But then I decided to shut it down and work for developers, especially those from remote rural areas. I could see that people from Tier II and Tier III cities do not get the kind of opportunities people in metro cities get. I started building a community of such people in 2016, which now numbers 15,000. While programming was my prime skill, I had to develop many more skills like communicating with the target audience as well as some skills in sales that could help me run an organisation.
Q. Is Jina AI your first venture into open source?
A. No. I built my own open source project back in 2016-17. I started teaching people about JSON Web Token (JWT) and what is JSON Web authentication. I did this through interactive demos. I started this developer community, and then we needed a package for encrypting some of our data. So I created an NPM package and published it on GitHub.
I had organised an online conference called the Git Commit Show in the pre-Covid days. I open sourced all the tools we had used to organise that online conference, and got a huge response from the audience. That built my confidence and indicated that open source is a great way to engage and collaborate with developers.
Q. What is meant by neural search?
A. When you search for something on Google with a keyword, it will probably give you results based on that or something similar to it. But searching by keywords is not the only way.
We use neural networks to do a semantic search, which makes your experience so much better. It doesn’t matter what you provide – keywords, images or videos — it will give all the relevant results. This search is based on the examples you provide, and it learns from that. So the next time you search for something, it will just look into the repository that you have built to give you the results.
Q. What are the challenges you have faced in your open source journey?
A. Courage is needed to be more open and transparent, and to share your code with people. One also needs a change in mindset and loads of confidence. I often felt that my code may not be perfect, and was not sure if I should share it with people. There is also a fear about how secure the code is.
However, when I had organised an open source event, people like Daniel Stenberg and Sir Tim Berners-Lee encouraged me to just go ahead and open source the code, since people would support the project and also point out vulnerabilities and help to fix them. The project would become more secure that way.
I remember a recent incident where community members and developers teamed up for an online meeting to publicly discuss queries and concerns on Jina AI. We started this call and someone Zoom-bombed us by playing very loud music along with verbal racist abuse. This was a pretty big challenge. Thankfully, our community supported rather than blamed the organisers.
Ensuring the safety of my community members is a mindful task, but the good generally outweighs the bad.
Q. How can open sourcing a project speed up its development?
A. Open sourcing does help to speed up the project. First, once you find members, you get the motivation to take the project forward faster, and that’s what happened with me at Jina. There may be people who challenge you every day, and tell you that they have found some problem with the project and it’s not working. So you need to build the motivation, internally and externally. But then there are those who ship their code quickly, helping the project move forward.
More importantly, collaboration works magic mysteriously. When you share the project with the world, people start using it, and then they come up with their own ideas and notions as to how it can be used and build something further out of it. They may build something that is a little different from what you built, and someone else may even make it better. And this eventually grows your project at a very exponential scale. Such growth cannot happen if you are working alone, limiting that code to a certain number of people in your team.
The best example I can share is from Jina itself. The first version took us about one year to release and the next version was released in three months. Our contributors have grown to maybe around 200 or more.
Q. Do you mean to say that the number of people working on it can also impact a project’s speed?
A. It is about the number of people who are engaging with the project and not necessarily only the ones contributing code to it. At the time of the Jina 1.0 release, we had 2000 stars on GitHub. This figure has reached 13,500 now. So these many people are sharing their ideas on Slack, reporting issues or helping out with other stuff that they find new.
For instance, the Jina 1.0 version puts structure to the data, creates a pipeline to operate on this data, and then eventually makes it available to the audience for search operations. The version looked very simple initially, but we figured it was complex. So we decided to break it down into three separate projects and saw an increase in its usage. The growth of Jina has been exponential in 2021.
Q. How does one sustain open source projects? Any tips?
A. As a project lead, always remember you have to lead the project. Do not expect people to come and start contributing on the first day. You have to put in the effort as a leader and have to move the project forward. There will be contributions to the code or in fixing vulnerabilities, but there is no one who will give so much support that the maintainer or lead does not need to think about the project.
You can think of how to make people come in and contribute to your open source project. A couple of things have worked well for me. When you are passionate about your project and people recognise that, the chances that they will get inspired and start supporting you are very high.
The other thing is to keep your project simple. If people find your project really complex to understand, they are doubtful about whether they can contribute to the code. So make your project easy to contribute to, and cut down the clutter in terms of code and ideas.
Q. Who do you look up to as inspiration?
A. Tim Berners-Lee, the creator of the Web, is my first inspiration. I keep communicating with him and sharing emails, and I do get a response once in a while. The other person is Daniel Steinberg, the author of cURL. He built that project, which is installed on every computer in the world. So that is something that is very inspiring. He is very transparent and shares his experiences on his blog. I seek advice from him, and I think he is among the best persons to follow in the open source world.
I would encourage everyone to attend events, as that is where we can come across the achievers in the open source industry. That really helps to build your network and push projects forward.
Q. Any nervousness during your open source journey, and how did you overcome it?
A. Oh, yes! That happens almost every once or twice a year and it’s absolutely fine to accept those situations. I had to shut down my startup when I realised it was not working out. That was really difficult, but it kept me pushing to think about what exactly I wanted to do.
I moved on to working with Jina. I now understand that one need not go by the trends out there, but rather be more focused on what you really want to do for the long term. We have a huge community of 15K ML engineers at Jina now.
Q. What’s your message to all the developers out there?
A. I think people should be more aware of open source as I can see a rapid transformation in the industry. I strongly believe that any software that is proprietary right now will soon have an open source alternative — if not this year, then maybe in just a couple of years. I foresee a future built on open source.
If you are a developer, this is the time to think about open source and the ways in which you can contribute to it and be more involved in that space. Just keep learning and build projects. And the bigger question to address now is building sustainable open source projects. There is a sweet spot for developers to build open source software in a transparent manner and make it sustainable by producing a software as a service (SaaS) version of it. For those who do not want to go through the struggle of maintaining software, SaaS could be the best solution. This may be a bit early to state, but I am pretty confident that most of the projects we see in open source will really move to this kind of model to make it sustainable. And that’s the future — SaaS plus open source.
Q. Are you planning any expansion and what are the skills you are looking for?
A. At Jina AI, we are currently hiring for engineering roles. For my team, I am looking for people with technical writing, community development and event management skills
