#########################################
# STRUCTURED DYNAMIC DATA GENERATOR v.1
#----------------------------------------
# Can parse TEA Data Templates(TDT)
# and generate random data based on
# specification and embedded TEA code.
# Details on TDT syntax in the TEA TAZ Book
#########################################

#----| Sample Data Specification |------
#in case no explicit data was provided, use TDT template...
i:{ [{ "Name": "<p!:8|t!.:|z*:>", "Age": "<n:26:8>" "Description": "<p!:|s:|s:|z:>" },][5] }

#----| CONSTANTS |------
v:vOP_LT:{<}
v:vOP_INCR:{+1}

#----| PRE-PROCESS |----
v:vSCHEMA #store the data schema

#extract the core schema object only...
d:\[\d+\] | d:[\[\]] | v:vSCHEMA_TEMPLATE

#extract the number of objects to generate..
y:vSCHEMA | d!:\[\d+\] | d:[\[\]] | v:vN_RECORDS

#initialize empty output dataset..
v:vDATA_OUTPUT:{}

#init counter of records created thus far
v:vN_ITER:0

#----| DATA-GENERATION-PROCESS |----
l:lSTART_GEN

#test if we already reached limit, so we finish...
g*:{}:vN_ITER:vOP_LT:vN_RECORDS | r.: | f:^false$:lFINISHED_GEN

#process record to be added to list..
y:vSCHEMA_TEMPLATE | v:vGEN_RECORD

#######| PROCESS a RECORD |#####
l:lPROCESS_RECORD
y:vGEN_RECORD

#----| inject dynamic values... |----
#first, get all dynamic expressions
d!:<[^<>]*> | v:vDYN_EXPRESSIONS

y:vDYN_EXPRESSIONS
#if empty, move on...
f:^$:lDONE_PROCESS_RECORD

#extract only the first one...
d:><.* | d:< | d:> | v:vTEA_EXP 

#evaluate it and store result
e*:vTEA_EXP | v:vTEA_EXP_VAL
#override dynamic field in template with computed value
y:vTEA_EXP | x:< | x!:> | v:vTEA_EXP | r*:vGEN_RECORD:vTEA_EXP:vTEA_EXP_VAL

#store final generated record
v:vGEN_RECORD

#process the remaining expressions too...
j:lPROCESS_RECORD

#done processing record template
l:lDONE_PROCESS_RECORD

#add it to the list...
g*:{
}:vDATA_OUTPUT:vGEN_RECORD | v:vDATA_OUTPUT

#increment record gen counter
g*:{}:vN_ITER:vOP_INCR | r.: | v:vN_ITER

#loop
j:lSTART_GEN

#----| DATA-GENERATION-COMPLETE |----
l:lFINISHED_GEN

#post-process data... enclose into JSON array...
y:vDATA_OUTPUT | x:{[} | x!:{]} | v:vDATA_OUTPUT 

y:vDATA_OUTPUT #present final dataset and quit

<p!:4:{⬜}:{⬜⬛}>[5]

[
⬛⬜⬛⬜⬛⬜⬜
⬛⬜⬛⬜⬛⬜⬜
⬛⬜⬛⬜⬜⬜⬛
⬛⬜⬛⬜⬛⬜⬜
⬜⬜⬛⬜⬛⬜⬛]

[
⬛⬜⬜⬜⬛⬜⬛
⬛⬜⬛⬜⬛⬜⬜
⬛⬜⬜⬜⬛⬜⬛
⬜⬜⬜⬜⬜⬜⬜
⬛⬜⬛⬜⬛⬜⬜]

12,24,95,18,85,27,97,39,80,20,65,24,93,82,85,81,52,62,20,89

A Bar-GRAPH Generator
i:{12, 24, 95, 18, 85, 27, 97, 39, 80, 20, 65, 24, 93, 82, 85, 81, 52, 62, 20, 89} #some sample data
v:vSEQUENCE #stash comma-del sequence
#count the values...
d:[ ] | d!:{,}| v: | v!: | x!:{+1} | r.: | v:vNSEQ

#get the word MSS
y:vSEQUENCE | h:, | d:, | u: | v:vSEQ_MSS
#but we would rather present a graph with items sorted based on order
#thus, override MSS with sorted one
#y:vSEQ_MSS | o: | v:vSEQ_MSS

#initialize graph structure as empty graph
v:vGRAPH:{}

#-----| For each item in the MSS, obtain its frequency |---
l:lBUILD_GRAPH
y:vSEQ_MSS
d:[ ].*$ | v:vITEM #first, get the item
#build a lookup pattern [,]?N[,]?
y:vITEM | x:{[,]?} | x!:{[,]?} | v:vITEM_REGEX
#use it to reduce original sequence to only instances of item..
y:vSEQUENCE | d*!:vITEM_REGEX
#remove extraneous commas
r!:[,]+:, | d:^, | d:,$
#count instances of item then..
d!:{,}| v: | v!: | x!:{+1} | r.: | v:vNITEM
#build the visual for item for the graph structure
#N:--...N-times
v:vGRAPH_ELEMENT:{-}
v:vGRAPH_ITEM:{} | x*!:vGRAPH_ITEM:vITEM | x!:{:} | v:vGRAPH_ITEM
v:vN_ITEM_GRAPH_LEN:{0}
l:lSTART_ITEM_GRAPH
x*!:vGRAPH_ELEMENT:vGRAPH_ITEM | v:vGRAPH_ITEM
y:vN_ITEM_GRAPH_LEN | x!:{+1} | r.: | v:vN_ITEM_GRAPH_LEN
#build and run test...
y:vN_ITEM_GRAPH_LEN | x!:{==} | x*!:vNITEM | r.: | f:true:lDONE_ITEM_GRAPH:lSTART_ITEM_GRAPH
l:lDONE_ITEM_GRAPH
#add item graph to main graph...
y:vGRAPH | x!:{
}|x*!:vGRAPH_ITEM | v:vGRAPH #update graph
#remove this item from the MSS, and then loop...
y:vSEQ_MSS
d!:[ ].*$ | t!.: | v:vSEQ_MSS #first, get the item
#if mss empty, finish, otherwise loop..
f:^$:lGRAPH_READY:lBUILD_GRAPH
l:lGRAPH_READY
y:vGRAPH #present final graph :)

:-
24:--
85:--
20:--
12:-
95:-
18:-
27:-
97:-
39:-
80:-
65:-
93:-
82:-
81:-
52:-
62:-
89:-

The DATA ANALYSIS:
====================
RANGE:12-97 | MODE:24 | MEDIAN: | MEAN:57.5 | VARIANCE:922.85 | STANDARD DEVIATION:30.378446306550966 | MODAL SEQUENCE:24 85 20 12 95 18 27 97 39 80 65 93 82 81 52 62 89
====================
The DATA:
====================
12,24,95,18,85,27,97,39,80,20,65,24,93,82,85,81,52,62,20,89
====================
Finished summarizing the data using measures and statistics.

The DATA ANALYSIS:
====================
0:-----
1:----
2:----
3:-----
4:-
5:--
6:---
7:----------
8:-------
9:-
10:-------
====================
The DATA:
====================
7,5,6,7,8,10,3,8,1,7,3,10,7,0,3,2,7,8,2,10,8,8,10,8,10,1,3,7,2,7,1,4,0,5,8,1,3,10,0,9,7,6,7,8,0,10,6,2,7,0
====================
Finished summarizing the data visually.

SAT v2
#########################################
# STATISTICAL ANALYSIS TOOL (SAT) v.2
#---------------------------------------
# An interactive program for analyzing
# numeric data using 3 analysis modes:
# descriptive/visual, exploratory and predictive.
#########################################

#first, pick any user/externally provided data...
v:vDATA

#====| CONSTANTS |====
v:vHLINE:{
--------------------
}
v:vHHLINE:{
====================
}
v:vPROMPT_METHOD:{Select Analysis Method: [1] Descriptive | [2] Exploratory | [3] Predictive
}
v:vANALYSIS_MSG:{No Analysis Done Yet!} | y:vANALYSIS_MSG | v:vRESULT
z.:PLATFORM | v:vPLATFORM #store underlying platform/environment...

#in case no data was presented, use hardcoded dataset...
y:vDATA
i:{12,24,95,18,85,27,97,39,80,20,65,24,93,82,85,81,52,62,20,89} | v:vDATA
j:lPREPROCESS_DATA

#prompt for data...
l:lPROMPT_FOR_DATA
i!:{Please Enter [qQ] to Quit, otherwise a [space or] comma-delimited sequence of numbers:} | i*: | v:vDATA

#----| should we quit? |---
f:^[qQ]$:lQUIT

#----| pre-process and clean data |---
l:lPREPROCESS_DATA
g: #eliminate white-space
r:,]$:] #fix final comma from some TDT generators
d!:[\d ,] #apply filter
g:,:[ ] | r!:,,*:, #enforce strict , delimiter
v:vDATA_CLEAN #for presentation
x:,|x!:, #to ease lookups
v:vDATA #override

#----| ensure data conforms to structure required otherwise flag error |---
f!:^,\d+(,\d+)*,$:lERROR_INVALID_DATA:lVALID_DATA

l:lERROR_INVALID_DATA
i!:{SORRY, but DATA entered was INVALID.} | i*:
j:lPROMPT_FOR_DATA # [re-]prompt for correct data

#we have proper data, might proceed...
l:lVALID_DATA

#----| present clean dataset and prompt for analysis method |---
l:lANALYSIS_PROMPT
y:vDATA_CLEAN | x*:vHHLINE | x:{The DATA:} |x*!:vHHLINE | x*!:vPROMPT_METHOD | v:vPROMPT | i*: | v:vANS


#check if we got meaningful answer, otherwise re-prompt
t!.: | f:^$:lANALYSIS_PROMPT | f!:^[123]$:lANALYSIS_PROMPT

#process response then...
f:1:lPROCESS_ANALYSIS_DESC | f:2:lPROCESS_ANALYSIS_EXPL | f:3:lPROCESS_ANALYSIS_PRED

####| START PROCESSING ANALYTICS |####
v:vANALYSIS_RESULT:{}


#----| perform visual/descriptive analysis |---
l:lPROCESS_ANALYSIS_DESC

y:vDATA
v:vSEQUENCE #stash comma-del sequence
#count the values...
d:[ ] | d!:{,}| v: | v!: | x!:{+1} | r.: | v:vNSEQ

#get the word MSS
y:vSEQUENCE | h:, | d:, | u: | t.: | v:vSEQ_MSS
#but we would rather present a graph with items sorted based on order
#thus, override MSS with sorted one
y:vSEQ_MSS | o: | v:vSEQ_MSS

#initialize graph structure as empty graph
v:vGRAPH:{}

#-----| For each item in the MSS, obtain its frequency |---
y:vSEQUENCE | r!:{,}:{,,} | v:vLOOKUP_SEQUENCE #for use in graph computations
l:lBUILD_GRAPH
y:vSEQ_MSS

d:[ ].*$ | v:vITEM #first, get the item
#build a lookup pattern (,|[^0-9])N(,|[^0-9])
y:vITEM | x:{(,|[^0-9])} | x!:{(,|[^0-9])} | v:vITEM_REGEX
#use it to reduce original sequence to only instances of item..
y:vLOOKUP_SEQUENCE | d*!:vITEM_REGEX
#remove extraneous commas
r!:[,]+:, | d:^, | d:,$
#count instances of item then..
d!:{,}| v: | v!: | x!:{+1} | r.: | v:vNITEM
#build the visual for item for the graph structure
#N:--...N-times
v:vGRAPH_ELEMENT:{-}
v:vGRAPH_ITEM:{} | x*!:vGRAPH_ITEM:vITEM | x!:{:} | v:vGRAPH_ITEM
v:vN_ITEM_GRAPH_LEN:{0}
l:lSTART_ITEM_GRAPH
x*!:vGRAPH_ELEMENT:vGRAPH_ITEM | v:vGRAPH_ITEM
y:vN_ITEM_GRAPH_LEN | x!:{+1} | r.: | v:vN_ITEM_GRAPH_LEN
#build and run test...
y:vN_ITEM_GRAPH_LEN | x!:{==} | x*!:vNITEM | r.: | f:true:lDONE_ITEM_GRAPH:lSTART_ITEM_GRAPH
l:lDONE_ITEM_GRAPH
#add item graph to main graph...
y:vGRAPH | x!:{
}|x*!:vGRAPH_ITEM | v:vGRAPH #update graph
#remove this item from the MSS, and then loop...
y:vSEQ_MSS
d!:[ ].*$ | t!.: | v:vSEQ_MSS #first, get the item
#if mss empty, finish, otherwise loop..
f:^$:lGRAPH_READY:lBUILD_GRAPH
l:lGRAPH_READY
y:vGRAPH #present final graph :)
#affix graph to result
k!:^[ ]*$ #kick-out blank lines
v:vANALYSIS_RESULT

#what did we do?
v:vANALYSIS_MSG:{Finished summarizing the data visually.}

j:lDONE_PROCESSING #goto present results...


#----| perform numerical/exploratory analysis |---
l:lPROCESS_ANALYSIS_EXPL 
v:vRANGE:{}
v:vMODE:{}
v:vMEDIAN:{}
v:vMEAN:{}
v:vVARIANCE:{}
v:vSDEVIATION:{}
v:vMSS:{}

y:vDATA
v:vSEQUENCE #stash comma-del sequence
#count the values...
d:[ ] | d!:{,}| v: | v!: | x!:{+1} | r.: | v:vNSEQ

#get the word MSS
y:vSEQUENCE | h:, | d:, | u: | t.: | v:vSEQ_MSS | v:vMSS

#store data sequence we'll use in external functions
y:vSEQUENCE | d:^,:,$ | x:[ | x!:] | v:vARRAY_SEQUENCE

#compute the statistics...

#===| compute RANGE |
#get the word MSS sorted numerically
y:vMSS | o: | v:vMSS_SORTED
y:vMSS_SORTED | d:[ ].*$ 
v:vITEM_MIN #store smallest value
y:vMSS_SORTED | m: | d:[ ].*$  
v:vITEM_MAX #store largest value
y:vITEM_MIN | x!:{-} | x*!:vITEM_MAX | v:vRANGE

#===| compute MODE|
#get the normal word MSS
y:vMSS | d:[ ].*$ 
v:vMODE #store most frequent value

#===| compute MEDIAN|
y:vPLATFORM
f:WEB:lWEB_MEDIAN:lCLI_MEDIAN
l:lWEB_MEDIAN #to compute median via JavaScript
y:vARRAY_SEQUENCE
v:vCMD:{const median = arr => (sorted => (sorted[sorted.length >> 1] + sorted[(sorted.length - 1) >> 1]) / 2)([...arr].sort((a, b) => a - b));median(JSON.parse(AI));}
z*!:vCMD
j:lMEDIAN_READY

l:lCLI_MEDIAN #to compute median via Python

i!:{python3 -c "import json; AI='}
x*!:vARRAY_SEQUENCE
x!:{'; a=json.loads(AI); print(sorted(a)[len(a)//2] if len(a)%2 else sum(sorted(a)[len(a)//2-1:len(a)//2+1])/2)"}
v:vCMD | z*:vCMD
j:lMEDIAN_READY

l:lMEDIAN_READY
v:vMEDIAN


#===| compute MEAN|
y:vPLATFORM
f:WEB:lWEB_MEAN:lCLI_MEAN
l:lWEB_MEAN #to compute mean via JavaScript
y:vARRAY_SEQUENCE
z:{const mean = arr => arr.reduce((sum, val) => sum + val, 0) / arr.length;
mean(JSON.parse(AI));}
j:lMEAN_READY

l:lCLI_MEAN #to compute mean via Python

i!:{python3 -c "import json; AI='}
x*!:vARRAY_SEQUENCE
x!:{'; a=json.loads(AI); print(sum(a)/len(a))"}
v:vCMD | z*:vCMD
j:lMEAN_READY

l:lMEAN_READY
v:vMEAN


#===| compute population VARIANCE|
y:vPLATFORM
f:WEB:lWEB_VARIANCE:lCLI_VARIANCE
l:lWEB_VARIANCE #to compute variance via JavaScript
y:vARRAY_SEQUENCE
z:{const variance = arr => (m => arr.reduce((s, x) => s + (x - m) ** 2, 0) / arr.length)(arr.reduce((a, b) => a + b, 0) / arr.length);
variance(JSON.parse(AI));}
j:lVARIANCE_READY

l:lCLI_VARIANCE #to compute variance via Python

i!:{python3 -c "import json; AI='}
x*!:vARRAY_SEQUENCE
x!:{'; a=json.loads(AI); m=sum(a)/len(a); print(sum((x - m)**2 for x in a)/len(a))"}
v:vCMD | z*:vCMD
j:lVARIANCE_READY

l:lVARIANCE_READY
v:vVARIANCE


#===| compute STANDARD DEVIATION|
y:vPLATFORM
f:WEB:lWEB_SDEVIATION:lCLI_SDEVIATION
l:lWEB_SDEVIATION #to compute standard deviation via JavaScript
y:vVARIANCE
z:{Math.sqrt(Number(AI));}
j:lSDEVIATION_READY

l:lCLI_SDEVIATION #to compute standard deviation via Python

i!:{python3 -c "import math; print(math.sqrt(}
x*!:vVARIANCE | t.:
x!:{))"}
v:vCMD | z*:vCMD
j:lSDEVIATION_READY

l:lSDEVIATION_READY
v:vSDEVIATION


i!:{RANGE:} | x*!:vRANGE | x!:{ | MODE:} | x*!:vMODE | x!:{ | MEDIAN:} | x*!:vMEDIAN | x!:{ | MEAN:} | x*!:vMEAN | x!:{ | VARIANCE:} | x*!:vVARIANCE | x!:{ | STANDARD DEVIATION:} | x*!:vSDEVIATION | x!:{ | MODAL SEQUENCE:} | x*!: vMSS   
v:vANALYSIS_RESULT

#what did we do?
v:vANALYSIS_MSG:{Finished summarizing the data using measures and statistics.}
j:lDONE_PROCESSING #goto present results...


#----| perform numerical/predictive analysis |---
l:lPROCESS_ANALYSIS_PRED

y:vDATA
v:vSEQUENCE #stash comma-del sequence

#===| compute probabilistic extrapolation...|
# in our list extrapolation methods here, we are going to
#assume the dataset conforms to a normal distribution
#and so we shall do basic probabilistic prediction of
#the next value past the end of the dataset given 
#the nature of the sample (its mean and standard deviation)
y:vSEQUENCE | d:^,:,$ | x:[ | x!:] | v:vARRAY_SEQUENCE

y:vPLATFORM
f:WEB:lWEB_PREDICTION:lCLI_PREDICTION
l:lWEB_PREDICTION 
y:vARRAY_SEQUENCE
v:vCMD:{const nextProbable = arr => {
  let m = arr.reduce((a, b) => a + b, 0) / arr.length;
  let s = Math.sqrt(arr.reduce((a, x) => a + (x - m) ** 2, 0) / arr.length);
  return m + s * (Math.random() * 2 - 1);
};nextProbable(JSON.parse(AI));}
z*!:vCMD
j:lPREDICTION_READY

l:lCLI_PREDICTION 
i!:{python3 -c "import json, random, math; AI='}
x*!:vARRAY_SEQUENCE
x!:{'; a=json.loads(AI); m=sum(a)/len(a); s=math.sqrt(sum((x-m)**2 for x in a)/len(a)); print(m + s * (random.random()*2 - 1))"}
v:vCMD | z*:vCMD
j:lPREDICTION_READY

l:lPREDICTION_READY
v:vPREDICTION

i!:{Assuming Normal Distribution, NEXT value past end of dataset: } | x*!:vPREDICTION |
v:vANALYSIS_RESULT

#what did we do?
v:vANALYSIS_MSG:{Finished extrapolating the data.}

l:lDONE_PROCESSING
#present results...
y:vANALYSIS_RESULT |  x*!:vHHLINE | x!:{The DATA:} |  x*!:vHHLINE |
x*!:vDATA_CLEAN | x*:vHHLINE | x:{The DATA ANALYSIS:} |x*!:vHHLINE | x*!:vANALYSIS_MSG | v:vPROMPT | v:vRESULT | i*: | j:lPROMPT_FOR_DATA #not yet finished ;)

################################
#Finally, Quit, gracefully...
################################
l:lQUIT
y:vRESULT

1. This tool works well on WEB and Commandline.
2. Shall help you summarize your data visually.
3. Shall help you analyze the data via standard statistics.
4. Shall help you extrapolate/predict beyond your data.

2, 4, 8, 16

The DATA ANALYSIS:
====================
Computed Non-Linear Model: y = 0.3333*t^3 + -1.0000*t^2 + 2.6667*t + 0.0000
The predicted value at t = 5 is 30.00, and so that, with a value of r2 = 1.0000, this model explains ~100% of the data.
====================
The DATA:
====================
2,4,8,16
====================
Finished extrapolating the data.

The DATA ANALYSIS:
====================
Computed Linear Model: y = 4.6t + -4 
The NEXT value at t=5 is 19
The R2 (coefficient of determination) 0.92 means this model explains only ~92% of the data.
====================
The DATA:
====================
2,4,8,16
====================
Finished extrapolating the data.

The DATA ANALYSIS:
====================
Computed Non-Linear Model: y = 0.0833*t^3 + -0.3929*t^2 + 1.5238*t + -0.2000
The predicted value at t = 6 is 12.80, and so that, with a value of r2 = 0.9995, this model explains ~100% of the data.
====================
The DATA:
====================
1,2,3,5,8
====================
Finished extrapolating the data.