CodeLani

Hello, world…year 3!

2019-11-15T02:00:00-10:00

I started CodeLani 2 year ago today!

Since that time:

In the past year, CodeLani has gone from less than 1,400 languages to over 3,800! Almost a 3x increase!
In the past year, I added the programming language patterns page which now lists 90 patterns!
I also added the file extensions page which lists over 1,300 file extensions!
The programming language design conference page has 6 conferences listed for 2020.

Unfortunately the number of articles on CodeLani is 14 (counting this one). Just like last year, limited time and resources has meant building up the dataset took priority. Hopefully the next year will see a lot of growth in number of posts!

How many major global programming competitions are there?

2019-09-04T00:00:00-10:00

I have done many problems on Project Euler but I’ve never participated in competitive programming. The other day I got curious, what does the competitive programming landscape look like?

I started my quest by watching a great interview with a competitive programmer. Then I did some searching. Below are my results.

There are 16 major programming competitions

The first one started in 1970. A few of the newer ones are online only.

Most of them started or are hosted in the USA

The List

Here is my list. You can contribute to it on GitHub.

name	started	firstHostCountry	type
IMO	1959	Romania	Mathematics
ICPC	1970	USA	Programming
ACSL	1978	USA	Programming
IOI	1989	Bulgaria	Programming
ICFPProgrammingContest	1998	USA	Programming
TopCoder	2001	USA	Programming
TopCoderOpen	2001	USA	Programming
GoogleCodeJam	2003	USA	Programming
IEEEExtreme	2006	Online	Programming
CodeChef	2009	India	Programming
CodeForces	2010	Russia	Programming
AtCoder	2010	Japan	Programming
Kaggle	2010	USA	DataScience
FacebookHackerCup	2011	USA	Programming
HackerEarth	2012	USA	Programming
HackerRank	2012	USA	Programming

Gennady Korotkevich

I came across one interesting stat in my research. Gennady Korotkevich is a programmer who is currently the highest rated programmer on all 5 of the major competitive programming sites!

Future work

It would be interesting to see trends in the number of participants and prize amounts in these competitions over time. I’m curious if eSports with prizes in the millions, will sap people away from programming competitions.

How many programmers are there in the world?

2019-05-26T00:00:00-10:00

There are around 7.574 billion people in the world. How many of them are programmers?

The answer, of course, depends on how you define “programmer”. But before I define “programmer” and share my estimates, let’s look at what some other sources say.

What do other people say?

GitHub - 36 million developer accounts

As of May 2019 GitHub had 36 million developer accounts. Now, some of these are inactive, some are duplicates, and some are owned by people who signed up for GitHub but aren’t programmers. I myself have at least 2 GitHub accounts. Some programmers aren’t on GitHub–though I’m finding that to be increasingly rare.

If we guess that 30% of GitHub accounts are for non-programmers and 20% are duplicates, and that 90% of programmers have a GitHub account, we get an estimate of 20 million programmers in the world.

Evans Data Corporation

Evans Data Corporation estimated there were 23 million developers in 2018.

International Data Corporation

IDC says “there were 22.30 million software developers in the world at the outset of 2018”.

Daxx created some nice visualizations of both the IDC and Evans data.

What do I say?

I will define a programmer as someone who edits a text document in a programming language at least once a week, on average, in an IDE or text editor.

My early guess: there were between 10 and 30 million programmers in 2018.

Does every programming language have a central package repository?

2019-02-04T02:00:00-10:00

Like millions of other programmers, every day I depend on central package repositories (CR) like npm, PyPI and CRAN.

The other day I was curious: does every programming language have one of these? I decided to find out. I pointed my crawler and trained a model to check for a package repository for every one of the 3,006 languages I am tracking. The results surprised me.

★ Only 1% have them

My model found only 39 languages with central package repositories. (For comparison, Wikipedia lists ~20). That’s just ~1% of languages. I thought it would be higher.

★ ~30% of the Top 100 have them

Given that a programming language is very popular and appears in my top 100 list, it is about 15 - 30x more likely to have a CR. Given a language is not in the top 100, <1% will have a CR.

★ 2 Million+ Packages Total

There are over 2,000,000 packages (aka modules or libraries) across these CRs. That means there are about 1,000x more packages than there are programming languages.

With that many packages, name collision is certainly a problem (maybe a subject for another post), though not as much of a problem as in the domain system where 130,000,000+ “.coms” alone are registered.

★ The top 5 account for ~80% of all packages

At over 900,000, Javascript’s npm has almost more packages than all other CRs combined. Javascript, Java, PHP, Perl, and Python account for about 80% of packages.

★ GitHub has over 100,000,000 repositories

Given the size of GitHub, and it’s growth as somewhat of a universal central package repository (though totally unmoderated), and given that many (if not the majority) of the packages in these CRs are also listed on GitHub, it’s conceivable that GitHub is the largest CR and that the number of packages out there is easily 10x bigger than 2M.

★ Newer languages are not more likely to have CRs

This surprised me. The median age of a language with a CR is 24 (1995). Of the top 5 languages I mentioned above, all were created by then. Almost always the creation of the CR follows the launch of the language, sometimes by months or sometimes by years. I expected most CRs to be from newer languages, but that wasn’t the case. While some new languages like Rust and Julia have CRs, others like Go and Kotlin do not.

★ A Visual

★ The List

Here’s my list of the main central package repositories for languages that have them. I cut the list a bit to only include CRs with more than 100 packages available. As always, let me know if you spot any omissions or mistakes!

id	website	packages	appeared
javascript	http://npmjs.org	901,025	1995
java	https://search.maven.org/	266,776	1995
php	https://packagist.org/	211,636	1995
perl	https://www.cpan.org/	176,876	1987
python	https://pypi.python.org/pypi	167,097	1991
ruby	https://rubygems.org/	154,445	1995
csharp	https://www.nuget.org/	141,524	2000
swift	https://cocoapods.org/	57,000	2014
clojure	https://clojars.org/	23,459	2007
rust	https://crates.io/	22,486	2010
r	https://cran.r-project.org/	13,674	1993
haskell	https://hackage.haskell.org/	13,487	1990
matlab	https://www.mathworks.com/matlabcentral/fileexchange/	9,718	1984
erlang	https://hex.pm/	8,069	1986
tex	https://ctan.org/	5,649	1978
stata	https://www.stata.com/manuals/rssc.pdf	4,608	1985
smalltalk	http://smalltalkhub.com/	4,534	1972
powershell	https://www.powershellgallery.com/	4,382	2006
emacs-editor	https://melpa.org/	4,079	1976
dart	https://pub.dartlang.org/	2,751	2011
maple	https://www.maplesoft.com/applications/	2,650	1982
ocaml	https://opam.ocaml.org/	2,224	1996
lua	https://luarocks.org/	2,047	1993
julia	https://pkg.julialang.org/	1,906	2012
d	https://code.dlang.org/	1,498	2001
dynamo-visual-language	https://dynamopackages.com/	1,494	2011
haxe	https://lib.haxe.org/	1,303	2005
racket	https://pkgs.racket-lang.org/	1,122	1994
elm	https://package.elm-lang.org/	594	2012
coldfusion	https://www.forgebox.io	519	1995
nim	https://nimble.directory/	499	2008
spark	https://spark-packages.org/	441	1988
prolog	http://www.swi-prolog.org/pldoc/doc/SWI/library/prolog_pack.pl	275	1972
mathematica	http://packagedata.net/	210	1988

Notes: as Jay18001 pointed out, a few of these repositories serve packages for more than one language. Cocoapods => Objective-C, and Nuget => F# and other .Net langs. In this post I collapsed things so each repo only has 1 language.

Update: 8/26/2019. Multiple readers pointed out that my stat for Ruby was off by 10x. I used the # I found on this page (https://rubygems.org/gems), which turned out to be just the gems beginning with the letter A. I apologize for the mistake and am very grateful for the corrections.

Which programming languages use indentation?

2019-01-21T02:00:00-10:00

Python, as one of the top 10 programming languages in the world, is the most popular programming language that treats indentation as significant. In these languages, programmers indent their code blocks instead of using braces, brackets, or other visible characters.

I was curious about how common these languages were so I did some brief analysis to answer these questions:

How common is this type of language?
How many languages use significant indentation?
Are new indentation languages on the rise?

1) Fewer than 2% of programming languages have significant indentation

2) Only 36 out of 2,500 programming languages in my database use indentation

These languages use indentation:

abc, aldor, boo, buddyscript, cobra, coffeescript, csl, curry, elixir, f-sharp, genie, haml, haskell, inform, iswim, literate-coffeescript, livescript, madcap-vi, madcap, makefile, markdown, miranda, nemerle, net-format, nim, occam, org, promal, python, restructuredtext, sass, scss, spin, stylus, xl-programming-language, yaml.

According to my present database, languages that strip indentation are about 70x more common than languages that employ it.

My methods and models are not yet perfect and may have missed some indentation languages. If my models missed ~80% of the languages, that would still be a 10x difference between the two classes of languages. If you know of a missing language, please let me know.

3) The creation of indentation sensitive languages has ticked up, but not by much

While there have been some notable new languages using significant indentation in the 2000’s, some of the biggest new languages don’t, including Go, Swift, Rust, TypeScript, Julia and Kotlin.

Takeaway: According to my current data, there has not been a significant increase in new languages using significant indentation.

Future Work

I would like to expand upon this to help language designers answer the question: will adding significant indentation increase the odds of my language becoming successful? I will continue to invest in my data collection and mining efforts to ensure I am not missing accurate information about the lesser documented languages.

UPDATE: masonic on HackerNews pointed out that COBOL should be included on this list. The column position of tokens is indeed significant in COBOL (and FORTRAN too), so perhaps in the next update I’ll clarify this categorization to adhere more closely to the offside rule, or will include a broader definition of significant indentation that includes some of these additional languages

Hello, world…year 2!

2018-11-15T02:00:00-10:00

I started CodeLani 1 year ago today!

Since that time:

CodeLani has gone from less than 400 languages to over 1,300! A 3x+ increase!
CodeLani has gone from less than 1GB to over 140GB of raw data! A 100x+ increase!
CodeLani has gone from tracking a few dozen dimensions to over 1,350! A 30x+ increase!

Unfortunately the number of articles on CodeLani is 9 (counting this one). Limited time and resources has meant building up the dataset took priority. Hopefully the next year will see a lot of growth in number of posts!

When did GitHub become the place to host new programming language projects?

2018-11-06T02:00:00-10:00

Before GitHub started in 2008, the source code for nascent programming languages was stored in a variety of places. In the early days it was physical media; later on it was publicly accessible servers; and even later it started moving to online source control systems like self-hosted SVN servers or Sourceforce.

But nowadays new language creation happens on GitHub more than anywhere else. Of the 44 languages created in 2008 that I track, 7% were put on GitHub that same year. Last year it was over 50%.

This does not include languages that were created and then moved their source control to GitHub a year or more after their creation. I wanted to know, if you are starting a language, should you create the GitHub project right away? According to the data that is the popular thing to do.

In the future I hope to refine this list to include factors such as whether the language has a corporate sponsor and how that affects things. I also hope to add more missing languages to the database, as always.

Data used for this chart:

appeared,newLangs,repoOnGitHubSameYear,percent
1950,2,0,0
1951,2,0,0
1952,1,0,0
1953,1,0,0
1955,4,0,0
1956,3,0,0
1957,4,0,0
1958,5,0,0
1959,6,0,0
1960,10,0,0
1961,2,0,0
1962,2,0,0
1963,8,0,0
1964,9,0,0
1965,5,0,0
1966,9,0,0
1967,6,0,0
1968,9,0,0
1969,8,0,0
1970,17,0,0
1971,6,0,0
1972,10,0,0
1973,6,0,0
1974,10,0,0
1975,6,0,0
1976,14,0,0
1977,13,0,0
1978,8,0,0
1979,5,0,0
1980,18,0,0
1981,4,0,0
1982,9,0,0
1983,13,0,0
1984,11,0,0
1985,24,0,0
1986,23,0,0
1987,14,0,0
1988,18,0,0
1989,13,0,0
1990,18,0,0
1991,16,0,0
1992,16,0,0
1993,32,0,0
1994,19,0,0
1995,27,0,0
1996,37,0,0
1997,29,0,0
1998,17,0,0
1999,20,0,0
2000,26,0,0
2001,32,0,0
2002,20,0,0
2003,25,0,0
2004,33,0,0
2005,30,0,0
2006,33,0,0
2007,30,0,0
2008,44,3,7
2009,30,7,23
2010,19,3,16
2011,45,9,20
2012,28,10,36
2013,26,16,62
2014,38,19,50
2015,26,12,46
2016,27,10,37
2017,26,15,58
2018,7,4,57

Which countries produce the most programming languages?

2018-10-28T02:00:00-10:00

Each programming language was created in a place (or perhaps, in a set of distributed places) and I am attempting to collect that data.

Currently I have the country of birth for about 260 of the over 1,000 languages I track.

In my data so far, the US has birthed more languages than all other countries combined.

After the U.S. comes the UK, Canada, France, and China, in that order.

There could be bias in my dataset, of course, as I haven’t yet added country for most languages. Things will clarify as the database continues to mature.

Data used for this chart:

country,count
AU,2
BE,1
BR,1
CA,12
CH,9
DE,6
DK,1
FR,11
GB,23
IS,1
IT,2
JP,4
KR,1
NG,1
NL,4
NO,4
PL,1
PT,1
RU,4
SE,4
US,167

What letters do programming languages start with?

2018-10-25T02:00:00-10:00

As the chart below shows, the number of programming languages beginning with a certain letter varies as much as 10x by letter.

Below is the same list, sorted by popularity.

A curious observation: why are A and C both ~3x more popular than B?

Data used for these charts:

firstLetter,n
a,59
b,22
c,75
d,20
e,28
f,38
g,30
h,15
i,23
j,27
k,15
l,45
m,57
n,31
o,30
p,69
q,10
r,33
s,76
t,24
u,13
v,13
w,9
x,10
y,4
z,6

What is the growth in programming languages over time?

2017-12-09T00:00:00-10:00

As I build up my database of programming languages I hope to be able to answer questions like:

is the rate of new language creation accelerating, slowing down, or constant?
what spurs the creation of new programming languages?
can we predict how many new languages will be released next year? in 5 years? in 10 years?

At the moment I am tracking 533 computer languages and I currently have creation years for more than half of those, including for 271 of the most popular ones.

Here is a simple line graph of the cumulative number of languages I have by year.

I will wait until I have more data to do more serious analysis (for example: the dip in the growth rate toward the end is likely due to my incomplete dataset and not indicative of a trend change).

But until I can provide a more comprehensive analysis, I thought I’d share my first look.

Update (12/10/2017): a number of readers have pointed out that some of the languages on this list are not programming languages. Future posts like this will be sure to exclude those.

Data used in this article as csv:

language,releaseYear
plankalkul,1948
g-code,1950
flow-matic,1955
regex,1956
fortran,1957
lisp,1958
algol,1958
cobol,1959
ibm-rpg,1959
mad,1959
jovial,1960
sako,1960
si,1960
teco,1963
ascii,1963
apl,1964
basic,1964
pl-i,1964
simula,1965
bcpl,1966
mumps,1966
logo,1967
spss,1968
b,1969
gml,1969
pascal,1970
forth,1970
scheme,1970
isbn,1970
lse,1971
c,1972
prolog,1972
smalltalk,1972
intercal,1972
ml,1973
sql,1974
diff,1974
sed,1974
tcp,1974
z-notation,1974
lex,1975
yacc,1975
sas,1976
s,1976
awk,1977
bourne-shell,1977
idl,1977
m4,1977
datalog,1977
pearl,1977
modula-2,1978
tex,1978
x86,1978
dbase,1979
ada,1980
scribe,1980
udp,1980
karel,1981
maple,1982
postscript,1982
turing,1982
abap,1983
vhdl,1983
turbo-pascal,1983
matlab,1984
objective-c,1984
transact-sql,1984
verilog,1984
common-lisp,1984
rpl,1984
asn-1,1984
simulink,1984
cpp,1985
clipper,1985
stata,1985
ampl,1985
arm,1985
bibtex,1985
bison,1985
ia-32,1985
latex,1985
mips,1985
erlang,1986
labview,1986
oberon,1986
eiffel,1986
fjolnir,1986
gas,1986
gdb,1986
gnuplot,1986
mathcad,1986
object-pascal,1986
sgml,1986
perl,1987
sparc,1987
unicode,1987
wolfram,1988
mathematica,1988
octave,1988
tcl,1988
picolisp,1988
bash,1989
coq,1989
http,1989
z-shell,1990
haskell,1990
j,1990
scilab,1990
pl-sql,1991
python,1991
visualbasic,1991
qbasic,1991
antlr,1992
gzip,1992
opengl,1992
powerpc,1992
lua,1993
r,1993
k,1993
applescript,1993
brainfuck,1993
html,1993
pdf,1993
utf-8,1993
vba,1993
cilk,1994
racket,1994
delphi,1995
java,1995
javascript,1995
php,1995
ruby,1995
visual-foxpro,1995
cfml,1995
mysql,1995
qt,1995
vbscript,1996
ocaml,1996
asp,1996
css,1996
drakon,1996
jscript,1996
lilypond,1996
postgresql,1996
puredata,1996
rest,1996
squeak,1996
supercollider,1996
uml,1996
xml,1996
c--,1997
mmx,1997
modelica,1997
rdf,1997
rpm,1997
curl,1998
actionscript,1998
soap,1998
vml,1998
vvvv,1998
autoit,1999
asterisk,1999
netlogo,1999
tom-oopl,1999
unlambda,1999
xpath,1999
alice,2000
csharp,2000
cmake,2000
drupal,2000
geo-ml,2000
sqlite,2000
subversion,2000
wsdl,2000
xbl,2000
d,2001
visualbasicnet,2001
processing,2001
json,2001
nsis,2001
protobuf,2001
svg,2001
tinyc,2001
yaml,2001
scratch,2002
gosu,2002
jsharp,2002
restructuredtext,2002
textile,2002
groovy,2003
boo,2003
autohotkey,2003
q,2003
amqp,2003
llvmir,2003
sawzall,2003
whitespace,2003
wordpress,2003
scala,2004
asymptote,2004
markdown,2004
nginx,2004
owl,2004
solr,2004
tiddlywiki,2004
fsharp,2005
sagemath,2005
django,2005
git,2005
haxe,2005
linotte,2005
mercurial,2005
mql,2005
puppet,2005
rails,2005
powershell,2006
aws,2006
hexagon,2006
jquery,2006
ooxml,2006
sass,2006
scss,2006
vala,2006
wml,2006
yii,2006
apex,2007
clojure,2007
cuda,2007
cython,2007
linq,2007
lolcode,2007
thrift,2007
xcore,2007
xquery,2007
nim,2008
geojson,2008
pandas,2008
pig,2008
pure,2008
sparql,2008
umple,2008
go,2009
coffeescript,2009
less,2009
mongodb,2009
opencl,2009
redis,2009
rust,2010
azure,2010
risc-v,2010
dart,2011
kotlin,2011
elixir,2011
google-cloud,2011
schemaorg,2011
stripe,2011
swagger,2011
webgl,2011
julia,2012
elm,2012
qalb,2012
typescript,2012
asmjs,2013
ats,2013
crystal,2014
hack,2014
swift,2014
idris,2014
solidity,2014
graphql,2015
webassembly,2015

Reproducibility Notes: The data in this post comes from the CodeLani database. It is possible to build a continually updating version of this article that updates as more data comes in.