TensorFlow basics¶
TensorFlow is a Google library made specifically for neural networks. In this note, we will be introduced to some basics of this main library, as well as the probability module. The topics discussed are:
# pip install -U TensorFlow
import tensorflow as tf
tf.add(1,2)
tf.add([1.,2.], [3, 4])
One thing to note with tensorflow is the types need to be consistent within arrays. Cannot have 1and 1. in same array, vector, or matrix. (Well, this worked for me but not him..)
tf.add([1,2], [3,4])
tf.square(123)
tf.constant([[1,3,4],[5,6,7]])
tf.linalg.inv([[1.,2.], [3.,4.]])
TensorFlow automatically uses float32 types, but can be changed. 16 is the fastest. Just keep in mind when error pop up.
Optimization¶
opt = tf.optimizers.SGD(learning_rate=0.3)
Standard gradient deriviation will get an objective function, find a starting point, and follow the steepest decent down to the global minimum.
var = tf.Variable(2.0)
Variables are the counter-part to constants. Can also be scalars, vectors or matrices. Only difference is they can change, while constants can't.
with tf.GradientTape() as tape:
y = var**2 +1
tape.gradient(y, var)
loss = lambda : var**2 /1.5
loss()
opt.minimize(loss, [var])
var
var_vals, loss_vals = [], []
for _ in range(30): # underscore is a valiad variable name, but means idc,dont worry about it
opt.minimize(loss, [var])
var_vals.append(var.numpy())
loss_vals.append(loss())
import matplotlib.pyplot as plt
%matplotlib inline
plt.plot(var_vals)
plt.plot(loss_vals)
plt.legend(['var', 'loss'])
Data sets¶
Can be nice to use pandas for preprocessing. But in industry, its a good idea to stick to as few libraries as possible. Tensorflow data sets are made for speed, whereas pandas is more for visualizing.
ds = tf.data.Dataset.from_tensor_slices(list(range(12)))
for i in ds.map(tf.square).shuffle(2).batch(3):
print(i)
Want to try and do the data manipulations on the graphics card. No need to manipulate on CPU, then transfer over to GPU.
ds.reduce(0, lambda state, value: state+value)
state = 0
op = lambda state, value: state+value
for value in ds:
state = op(state, value)
state
ds.map(tf.square).reduce(0, tf.add)
TensorFlow Probability¶
We will use this for Bayesian analysis using tensorflow.
# pip install tensorflow_probability
import tensorflow as tf
import tensorflow_probability as tfp
Generate data¶
Random sample from Binomial with N=60, and $\theta$=0.6
tfd = tfp.distributions
dist = tfd.Binomial(total_count=60, probs=0.6) # to pull a sample from some random distribution
sample = dist.sample(1)
sample # happened to match his perfectly, but didnt have to!
Posterior Probability - Uniform prior¶
thetas = tf.linspace(start=0., stop=1., num=500) # have no idea really how these thetas look
# P(theta) ->
dists = tfd.Binomial(total_count=60, probs=thetas) # my hypothesis is this data follows a Binomial dist.
probs = dists.prob(sample) # P( datat | hypothesis)
sample /60 # sample=38, 60 observations
idx = probs > 0.01
plt.plot(thetas[idx], probs[idx])
plt.xlabel('$\\theta$')
plt.ylabel('likelihood')
None
Posterior sampling¶
Start by generating categorical given my probabilities.
?tf.random.categorical
log_prob = dists.log_prob(sample)
N = 10000 # want 10000 samples from our posterior, sample according to curve to hypothesis
tf.random.categorical(log_prob, N)
idx = tf.random.categorical([log_prob], N)
Will return an index (or indexes, depending on N) responding to the categories
tf.random.categorical(tf.math.log([[0.3, 0.7]]), 10)
From these indexes, i need to get back the hypothesis.
theta_sample = tf.gather(thetas, idx)
plt.hist(theta_sample)
The exact same as the curve we got above.
posterior_sample = tfd.Binomial(total_count=60, probs=theta_sample).sample(1)[0,0,:]
plt.hist(posterior_sample)
unique, idxs, counts = tf.unique_with_counts(posterior_sample)
plt.bar(unique, counts)
sample # peak matches sample
Does my original model make sense with these results? Here we can say yes, since we have a peak again around 35, and roughly the same distribution.