Formula for DEA productivity from population

[vorenhutz]

From the encyclopedia:

If you have extra population, they're essentially placed into "supporting" roles within your industrial base, though there's a diminishing return as the excess increases. They will increase your DEA Productivity, but less so as the total number of extra people increases.


Since all DEAs are affected by this, I thought it was fairly important to know how it works in a bit more detail.
The relevant variables appear to be current popuation and DEA Employment. I collected a whole bunch of data:

Population DEA Employment DEA Prod from Pop
5 14 40
5 11 47
10 19 55
9 15 60
... etc


Unfortunately there is a lot of noise due to rounding to the nearest population unit. The following graph shows all the data, and a model that fits it fairly well.
http://homepages.paradise.net.nz/gbs1/productivity.jpg
Here f = (Population)/(DEA Employment),
p = (DEA productivity from population).
The model is defined piecewise, the formula is different depending on the value of f:

0<f<=1 p = zf
1<f<=2 p = z + k(f-1)
2<f<=3 p = z + k + k/2(f-2)
3<f<=4 p = z + k + k/2 + k/4(f-3)
4<f p = z + k + k/2 + k/4 + k/8(f-4)

z and k are adjustable parameters.
The best fit (minimum sum of squared deviations) was when k = 35.3, z = 103.7.
As a test, I found more precise numbers for Population and DEA Employment on one of my planets. It had 16.331 population and 15.215 DEA employment. The above model predicts 106.3% DEA productivity from population; the planets screen shows 106%.

Implications for usefulness of DEA capacity upgrades
Total DEA output depends on capacity of the DEA, and DEA productivity from population (and other things, but they are independent of population). All capacity upgrades require employees. So you can increase the capacity, but this may actually decrease the productivity so much that total output is less. You don't have to worry so much about this effect if you're in the 'diminished returns' region - the increased capacity more than makes up for the small decrease in productivity.
Algebraically:
DEA output = konstant * capacity * productivity
(after capacity upgrade) o + o' = k * (c + c') * (p + p')
(subtracting)
o' = k * ((c + c')p' + c'p)
The critical point is where the change in DEA output (o') is zero, this happens when
(c + c')p' + c'p = 0 (since k can't be zero)
rearranging:
-p' / p = c' / (c + c')
If the fractional decrease in productivity [-p' / p] is less than the fractional increase in capacity [c' / (c + c')], then total output will increase. For typical DEAs, the fractional increase in capacity (assuming the lower tech capacity upgrades were built) ranges from 0.33 to 0.4:

c' c c' / (c + c')
0.5 1 0.333
1 1.5 0.4
1.5 2.5 0.375
2 4 0.333

These capacity increases are quite high. It's difficult to imagine a single DEA capacity upgrade also decreasing the planetwide productivity so much that it's not worth building.
An extreme example: building automated factories on an early colony. It has 2 population and 3 DEA employment. Automated factories cost 0.75 employment. Productivity before this capacity upgrade would be 69.1% , and after, 55.3%. The fractional decrease in productivity is 0.2, which is less than the fractional increase in capacity (0.33). Even in this extreme example, the capacity upgrade increases total output. At higher population, diminishing returns on productivity would ensure that capacity upgrades would be even more worthwhile.

[Orionator]

:up:

[Stochastic]

Outstanding work!

The exact effects of population overstaffing have been something that I've been wondering about for awhile.

[Da_Blade]

Haha, funny you found a different formula then i did quite a while ago :)

http://www.ina-community.com/forums/showthread.php?s=&threadid=287815&perpage=30&pagenumber=8

The only thing that now still mistifies me, is the unemployable population, since there is three kinds of population:

1. There's employed population, this is equal exactly to the DEA employment (the precise one), if there is more pop then employment.
2. Unemployed population
3. UNemployable population.

number 1 and 2 are the only ones that affect DEA productivity, number 3 i never bothered to really find out, but i do know it's dependant of the other two. I'm going to look into some more soon i think. Also, i think my formula is somewhat easier, and it is also very precise...

[Da_Blade]

OK, i managed to get the last bit of info i needed to compare my previously posted formula with yours (can't believe i never bothered to look for the formula for unemployed and unemployable population, it's dead-simple :))

Anyways, my full formula now states:


pop = population
emp = dea employment
o = (emp+(pop-emp)*0,4)/emp
f = pop/emp
p = dea productivity

if (f < 1) then p = f
else p = 2-1/(o*square_root(o))


Now, i've produced a table which compares values from a bunch of planets i have taken data from:

http://home.planet.nl/~baas0504/moo3/deapic.gif
ProdR = the DEA productivity as stated in the planets screen
ProdB = the DEA productivity calculated through my formula
ProdV = the DEA productivity calculated through your formula

Now, as you can see both our formula's are slightly off. As i stated in the original thread; DEA productivity is calculated using last turn's population, also the DEA employment number is a bit off. Sometimes the game's using data of the previous turn to calculate things, and sometimes data from the current turn, i ascribe my deviations to that.

If you look a bit closer to the numbers though, you will notice your numbers will sometime differ a lot from the actual DEA productivity, sometimes a little. Also it is higher one time, but lower the other. I think this is because you use a system which divides up the graph in different stages of a straight line, so it differs more sometimes, and less other times.

Your differences cannot be atributed to pop and economic growth differences because you value is sometimes higher and sometimes lower then the actual value, while in Moo3 the population and economy usually always grows, it never shrinks. Thus your differences cannot possibly be ascribed to the planets screen using different values from this turn and last turn to calculate the figure. My number is consistantly slightly higher then the reported one, meaning that it can easily be because it used values from different turns to calculate the number in the planets screen.

But the most convincing number for me, which imo invalidates your theory, is the 5th column. Your dea productivity produces a 228% dea productivity! Which is impossible, since your DEA productivity can never exceed 200%. The way my formula is set up is to accomodate just that: it can never exceed the 200%, only come closer and closer to it, but never even reach the full 200% (though rounding will make it 200% after awhile of course)

[Da_Blade]

took some more samples and contstructed a graph:

http://home.planet.nl/~baas0504/moo3/graph.gif

[Ron_Lugge]

:bulb:

Too much text / pictures / calculations.

However, one thing I saw was that you used a series of equations. I believe that graph may be modelable by a single equation, specificly a square root one (its the closest match to the displayed graph).

I'd manipulate the numbers, but I'm so tired I'd screw it up.

[vorenhutz]

p = 2-1/(o*square_root(o))

Ah, I didn't think of that functional form. That does seem to work just as well (even without adjustable parameters). Also it eliminates all those piecewise functions. Nice :)
Sometimes the game's using data of the previous turn to calculate things
I'll buy that.
Which is impossible, since your DEA productivity can never exceed 200%
Well, there is no way you can know this for sure, but I agree in principle; I've never seen it >200%.

[Da_Blade]

Well, actually i have come to good terms withyour funstion as well. I think if you keep up the piecewise functions beyond f=4 in the same pattern, you might actually end up with a far better function then mine. It's quite easily written in code, and because the divisors keep doubling, you never reach the 200% too. Because my main gripe with your function has been the high overstaffing results.

Now, if this kind of unending function was even remotely possible to make in Excel i'd already tried it out. Now i have to wait before i feel like whipping up a script to draw the unending piece function.

The problem has been that my functions is always just above the originals, and it struck me yesterday or something, it should get closer to the in-game displays the higher the overstaffing, because the population growth has little effect on the percentages. But in fact, my results lay higher then the actual displayed numbers the higher the number get. Now, i could correct this by using a mulitplier of course, but it's not very neat...

Ah well, in the end it doesn't really matter anyways, as long as we understand the general behavior of DEA productivity. WHich has been clearly shown now anyways :)

[Beamup]

Originally posted by vorenhutz
Ah, I didn't think of that functional form. That does seem to work just as well (even without adjustable parameters). Also it eliminates all those piecewise functions. Nice :)

A useful way to look at data to find such things is to plot it on three scales. First, a linear scale. Second, a log-log scale. Third, a semilog scale. If it's a straight line on the first, you can tell the form is (as far as your data shows) ax+b. If it's linear on the second, it's of the form ax^b. If it's linear on the third, it's of the form ae^bx. If it's not linear on any of the three, it's something more esoteric.

Definitions if you don't know:
linear scale: plot y vs. x
log-log scale: plot ln y vs. ln x
semilog scale: plot ln y vs. x

I would suggest looking at the data in the form ln (productivity) vs. ln (population) (or whatever form of population you're looking at, e.g. employable population) over the > 1 range. Do a linear regression (LINEST, in Excel) and check the slope and r-squared value. If the slope is .5 and the r-squared close to 1, that is very solid evidence that the function is truly a square root.

[DavidByron]

Now, as you can see both our formula's are slightly off

Call me when they are not!

Vorenhutz: Da Blade and I are old hands at this one, but we've never cracked it. Piece of advice: the numbers that are reported on the screen may not be corresponding with each other. This might have changed post-patch but certainly before the patch some of the population related figures appeared to refer to the next turn or the previous turn. This one-turn-off aspect might explain small discrepencies.

[Da_Blade]

Well, as i said in my post before, the discrepancies between turns would account for large differences in the lower stages of f (pop/employment), but in fact it gets more the higher f becomes. Since population has an increasing diminshing effect on dea employment this can only mean my formula is off....

Looking at the graph some more, i am more and more convinced vorenhurtz was on the right track: as long as you keep the division going on forever. It's an easy enough thing to program, and you won't even have to include the math libraries to do vorenthurtz' formula.

If you look at the graph you see that except for a few off readings, the real data seems to exhibit those "corners" too. So i officially abandon my formula and am now a vorenhurtz-believer ;), as long as you keep the division going for each "f", and i'd say the z and k need some tweaking.

For anyone interested in the code for calculating this (c++ and java compatible ;)):


pop = population
emp = dea employment
z ~= 1
k ~= 0.35
f = pop/emp
p = dea productivity

for(x=0; f > 0; x++)
{
if (x>0&&f<1);
{
p = p + (k/(2^(x-1)))*f;
}
else if (x==0&&f<1)
{
p = z*f;
}
else if (x==0&&f>=1)
{
p = z;
}
else
{
p = p + (k/2^(x-1));
}
f--;
}



The p at the end of the loop should give the dea employment.
I am no great programmer so i am sure it can be done more effeciently, but you should get the sense.
See that in mathmetical sense my formula looked more simple, but in programming sense vorenthurtz' formula is actually simpler. So you got my vote vorenthurtz, for the adjusted formula that is ;)

[Da_Blade]

OK, having recalculated vorenhurtz' formula into infinity i have come back to my own formula.

http://home.planet.nl/~baas0504/moo3/deagraph2.gif

[Beamup]

Please go ahead and plot the data on a log-log scale and do a linear regression on it, if you would. That really would go a VERY long way towards settling the question once and for all. I could do it myself, but you've already got all the data in your spreadsheet, so you could do it much faster.

[Da_Blade]

Well, honestly, i've had it it all in uni, but i forgot most of it and i certainly do not know how to do it in excel. Besides, i have tried a logarithmic scale, it's not a straight line on there... but neither is my formula.

[vorenhutz]

@Beamup
With some modifications, that procedure should work.
Da_Blade's formula simplifies to
p = 2-(0.4f + 0.6)^(-3/2)
for f>1.
So a plot of log(2-p) vs log(0.4f + 0.6) should be linear with slope = -3/2 and intercept = 0.
I don't have my data here, but I'll try this out later.

I did this and got a slope of -1.28 and intercept of -0.0171. This is near enough to slope = -4/3, intercept = 0.
Since I'm too lazy to generate another plot, here is a nice bandwidth-efficient table:

f p
1 1
1.2 109.8%
1.4 118.0%
1.6 124.9%
1.8 130.9%
2 136.1%
2.2 140.7%
2.4 144.7%
2.6 148.3%
2.8 151.5%
3 154.3%

p.s. this is still not completely right. There is a noticeable trend in the residuals with the above model (which is why the best fit straight line used a slightly lower slope and intercept).



Other than the R^2 value (goodness of fit) another thing to look for is systematic trends in the residuals (predicted - observed values). Systematic trends indicate that you're using the wrong functional form, even though you may have a good fit. For example it should be obvious from the residuals if the true function is really piecewise or not. (I thought it would be piecewise because I'd just been looking at production overdriving).

@Da_Blade
How did you work out the employment formula (o = 0.4f + 0.6). From the employment data on the demographics tab?

In any case, you so rarely see f>3 or so in a real game that it doesn't really matter what the function looks like in that region.

Any thoughts/feedback on my implications section?

[Beamup]

Originally posted by vorenhutz
With some modifications, that procedure should work.

Yeah, I was being sloppy and not detailed, I know. I'd have done it properly if I was doing it, but I wasn't. Anyway, I at least am convinced now.

[vorenhutz]

OK, I think I've nailed it. (But maybe I speak too soon ;) )
I collected some more data, this time I also noted the unemployment and employment figures in the demographics tab (and also used the more accurate population value from there). Data were from 16 planets on 2 consecutive turns [whopping big table follows]

Planet Population 2 DEA Employment 2 Productivity 2 Unemployment 2 Employment 2
Pi Draconis IV 170.77 173.55 52 53 159 159 26 26 31 31
Celtsi III 125.94 126.71 14 14 185 184 33 33 11 12
Pi Draconis V 121.82 123.6 51 52 146 147 22 22 43 42
Whynil IV 100.5 100.4 26 26 165 165 28 28 26 26
Tau Aquil VIII 99.01 99.75 44 44 145 145 21 21 45 45
Don IV 89.32 88.6 50 51 135 134 17 17 56 58
Pi Draconis VI 86.46 86.32 54 54 129 128 15 14 62 63
Scorpio II 86.32 86.12 44 44 138 138 19 19 51 51
Proxima V 72.78 72.98 44 45 129 129 15 15 62 62
Lacerta V 62.21 62.5 13 13 171 172 30 29 21 21
Whynil II 60.5 60.35 40 40 125 125 13 13 66 67
Biot III 59.78 60.01 43 43 121 121 11 11 72 72
Dendo IV 55.87 56.27 49 50 109 108 5 5 88 89
Nusakan I 53.16 52.77 41 41 116 116 9 9 78 78
Nusakan III 50 50.08 37 37 119 119 10 10 74 74
Nobi V 41.7 42.04 27 27 126 127 14 14 65 64


After playing around for a bit, I found that
emp% = emp / pop * 100%
unemp% = U * (1- emp/pop) * 100%

emp -> DEA Employment
pop -> Population
unemp% -> unemployment
emp% -> employment

I found U by linear regression, ~= 0.38. The correlations are better using data from the current turn, rather than last turn's values as suggested upthread.

Now for productivity. I was using the formula
p = 2 - (0.4f+0.6)^(-4/3)
where p is productivity and f = pop / emp.

I generalised this to
p = M - (Uf + (1-U))^-D

where now capital letters are adjustable parameters.
M is the maximum value of p (as f -> infinity)
U is the fraction of unemployed that are 'voluntary'
D establishes the rate of diminishing returns, i.e. How fast p approaches M as f increases.

I used excel's solver function to do the fitting. I got it to minimise the sum of squared deviations (between productivity ingame, and calculated according to the model) by adjusting M, U and D. (This is a nonlinear least squares fitting procedure, which is superior in some ways to a linear fit of linearised data).
What I was expecting to happen was solver would change the parameters maybe a tiny bit to get an incrementally better fit, and leave it at that. Good old solver surprised me :)
It threw up this:
M = 2.0
U = 0.64
D = 1.0

Which simplifies to a much nicer model, it doesn't involve square roots or piecewise functions:

p = 2 -1/(0.64*f + 0.36)

or (substituting f = pop / emp)

p = 2 - emp / (0.64pop + 0.36emp)

I'm not sure how to interpret this formula. I think it may be a typo in the code actually. Da_Blade's interpretation of his formula made a lot of sense. It should be like (employed pop) / (employed pop + unemployed pop), which would make the denominator of the ratio (0.36pop + 0.64emp) i.e. swapped around.

[Da_Blade]

Ah yes: much better fit, and away with that ugly square root! :)

That thing bothered me because they would have to include math libraries to make that formula. Well, i think we can finally lay this thread to rest, since noone will be interested in the maths, hehe...

Anyways, some last eye candy to enjoy, i collected data of nearly 50 grown planets to make the following graphs:
http://home.planet.nl/~baas0504/moo3/deagraph4.gif
prodV = Vorenthurtz' last formula
prodB = my old beast of a formula
prodR = dea productivities taken from the planets screen


http://home.planet.nl/~baas0504/moo3/deagraph3.gif
This is a graph where the difference for each of the planets' displayed data and the result of both formula's is depicted.

[Iskabis]

wtg, vorenhutz!!!!! very nice work! :up:

and nice graphs, blade. some of us like the math... :weird:
(i've been called weird often enough, i'm thoroughly accustomed to it now... :) )

-rhyssan

[Da_Blade]

Just one last note: i have found U to be 0.4 precisely actually. I'll explain:

unemp = (pop-emp)*0.4

This means, from the total population the amount of employment avaible on the planet is substracted first, this is the employed population. Now, from the rest, exactly 40% is labelled "unemployed", the remaing 60% is what i call "unemployable", think women (or men depending on cultrual values ;)), children, elderly etc.

Now, of course this inevitably leads back to your formula:

unemp% = ((pop-emp)*0.4)/pop = 0.4*(1-emp/pop)

This finally makes me believe the final formula is actually:
p = 2 - emp / (0.6pop + 0.4emp)

But, the exact values of those multipliers cannot be retrieved properly, since there is quite a lot of noise in the display of the dea productivity, i assume it's rounding errors at work in their best possible environment.

[DavidByron]

The multiplier for the unemploymnet calculation -- 0.38 or 0.40 or whatever -- is your tax rate (total of imperial, system and planetary) and thus varies from planet to planet.

Unemployment = (1-employment%)(tax%)
based on population, tax rate and employment requirements from the *previous* turn.

[DavidByron]

Had another look at this thorny problem.

First question in calculating population related formulae is to figure out which turn's figures you use in the calculation.

In the case of DEA productivity population, jobs total and the tax rate are all factors and the present turn's DEA productivity is calculated on the basis of last turn's numbers.

This is what throws off a lot of calculations straight away ---- the DEA productivity % matches LAST TURN's population, employment requirement, and DEA employment. (But this turn's unemployment %)

The emplyment % is an oddity because it is calculated by dividing last turn's jobs total by the present turns population. This means for a planet with falling population and less population than jobs emplyment % is actually over 100%. Because of this oddity the emplyment % is not helpful in calculating DEA productivity%.

At least this gets the right figures on the board. We want a relationship linking last turn's population jobs total and tax rate with this turns. To put it another way, if I if change tax delete a building or add/remove pop this turn, the effect will show up next turn on pop, jobs and tax rate, but only the turn after that on DEA productivity%.

Simplified partial formula.

With tax rate = 0% the formula seems to simplify to

DEA productivity = 2 - [jobs / population]

where [...] indicates rounding down to the integer. Increasing taxation decreases productivity%. An approximation for large populations is,

DEA productivity = 2 - (jobs / population) * (1 + tax rate)

[DavidByron]

It's quite close to being,

DEA productivity% = 2 - (employed% / 1-unemployed%)

(same formula as above when tax=0%)

If so then one way to think of it is to label that proportion of the population that is neither "employed" (directly in the DEAs) nor "uemployed" as something like "marginal workers".

Total pop = employed + unemployed + marginal

These marginal or semi-employed workers are spare population points that can be used to increase the output of DEAs. The bonus given (beyond 100%) is then simply

+ marginal / (marginal + employed) %

This formula seems to fit ok with the extremes of the observed results (including setting tax rate to 95%) to within a couple of %. Possibly the discrepencies are due to rounding and then using rounded figures in the next step, causing further rounding.

[Babylon4]

Ok - I think I actually understand this formula.

So Say I have a planet with 0% tax. 80% of my pop is employed, 10% is unemployed and 10% is neither which you label marginal.

My DEA productivity would then be:
2 - (80/(100-10)) + 10/(10+80) = 122%

This seems a little counterintuitive since a DEA with 100% employement would have only a productivity of 100% while a DEA with 50% employment and 50% unemployment would also have a productivity of 100%. But nothing with this game suprises me.

[DavidByron]

My DEA productivity would then be:
2 - (80/(100-10)) + 10/(10+80) = 122%

No. I gave 2 equivalent formulae and you combined them. it's either

2 - (80/(100-10)) = 1.11

or it's

+ 10/(10+80) = +.11
(that is +.11 added to the "normal" score of 1)

I am trying to find a really simple way to express the formula and I think

bonus = marginal / (marginal + employed)

gives you a sense of the "why" as well as being simpler. In words your bonus is equal to the proportion of your work force which is not already allocated to the basic needs of the DEA.

[DavidByron]

his seems a little counterintuitive since a DEA with 100% employement would have only a productivity of 100% while a DEA with 50% employment and 50% unemployment would also have a productivity of 100%.

It makes sense if you think of marginal workers as the spare workers you have after your minimal DEA needs are met. In both cases above you have zero marginal workers so obviously you get no bonus from them.

The "employment" figure is not really a total employment figure. That's why employment + unemployment isn't 100%. It's the employment% which is perhaps misnamed here. It's more a "proportion of population in core employment positions" and doesn't have much uselful meaning.

The actual employment% listed is even more bogus because it divides the number of population in core DEA jobs last turn by the population this turn (meaning you can have employment over 100% if your population is falling).

[Babylon4]

OK - reading - thinking - reading - thinking

The real key to having productivity over 100% is the marginal workers. In fact, as long as all your DEA's staffed at a minimal level the marginal workers - those neither employed nor unemployed - are the driving force behind a DEA's productivity.

The percentage of marginal workers should be in bold surrounded by gold stars!


unemp% = ((pop-emp)*0.4)/pop = 0.4*(1-emp/pop)

It would seem from this formula that the unemployed are 40% (or whatever the total of all taxes on the planet are, 40% being a rough estimate) of leftover population after DEA's are staffed, and marginal workers would be the rest. Is this a roughly accurate assumption? This would seem to be another good reasong for keeping system and empire taxes lower, it lets a larger percentage of nonemployed become marginal workers rather than unemployed.

BTW I'm really glad this thread was resurected, it has really enhanced the way I analyze my planets and taken a lot of the mystery out of how some of these numbers are generated.

[DavidByron]

The tax = 40% assumption depends on your playing style I suppose. I get about 38%.

Less tax means higher DEA productivity% but then higher tax is such a good thing....

The main impact this has on my thinking is in that is junks the theoretical max-taxation policy. Basically it goes like this in theory:

The unrest imperial grant reduces unrest on a linear basis. I think it's something like 3.8 AUs per point of unrest, but whatever it is it doesn't vary.

Now the amount that unrest increases by when you hike your taxes 1% is not a constant --- it increases as your tax increases. Initially it's about 6 unrest/region or quickly it's 12. Then it goes to 20 and finally 40.

But at some point your planet can be maing enough extra income from the 1% tax increase that you can afford to pay the increased unrest from the increased revenue and still make a profit. At this point theoretically you might as well tax at the highest possible rate. A sort of tax event horizon has been reached.

The problem is your DEA productivity % would be dramatically decreased by 80-95% taxation. Still you could probably boost taxation to around 50-60% without hurting the DEA productivity too much.

[Awsric Armitage]

Bump to the front page and please rate for easier searching for next 60 day bump.

Oh.

BUMP.

[Colt374]

Bumpity Bump