https://www.jackogrady.me/contact daily 0.75 2019-09-23 https://www.jackogrady.me/photovoltaic-battery-charger daily 0.75 2021-07-19 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1553756127556-6BBPBTD2GFMSEQVSVB8M/simple_system%40150x.png Photovoltaic Battery Charger Simple View The circuit has two halves, both of which rely on the Li-ion cells in the center. This figure shows a simplistic view of each major components. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1553756212106-8CZWIIS7YLWTCA3PVARH/full_system%40150x.png Photovoltaic Battery Charger Full View This diagram shows how the battery management system (orange boxes) interfaces with the photovoltaic battery charger. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1555448928931-M19OL61YBOO7SXTT55WA/suncharge_new%40150x.png Photovoltaic Battery Charger - Plugging into the Sun Last Updated: 3/6/19 With a fascination for renewable energy technologies, I figured the best way to explore would be to build my own. So, I made a photovoltaic battery charger to charge my phone with sunlight. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1538952581636-V4YS95V9384W7J3QLENI/IMG_6633.jpg Photovoltaic Battery Charger https://www.jackogrady.me/photovoltaic-battery-charger-nimh daily 0.75 2019-02-16 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1547860533803-CUAZFT6SXMZAJKJFOUZA/solar_charge_2.JPG Photovoltaic Battery Charger - NiMh - Plugging into the Sun Last Updated: 1/25/19 With a fascination for renewable energy technologies, I figured the best way to explore would be to build my own. So, I made a photovoltaic battery charger to charge my phone with sunlight. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1548024070617-43P37TU5LNCL1TUMZQWY/Arduino_ammeter.png Photovoltaic Battery Charger - NiMh The Arduino measures the voltage drop between A0 and GND, which are wired to the top and bottom of the resistor, respectively. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1547776516445-ILYZWM3M7NM4QD6K6CRZ/battery_discharge_curve.png Photovoltaic Battery Charger - NiMh The discharge curve provided by the battery manufacturer for Eneloop AA. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1548023583079-JKFV9Y9CLDE9EFVCAA8G/circuit_v2.png Photovoltaic Battery Charger - NiMh https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1547777413918-1H6BF2CEVD6JBNUYIB6G/excel_data_collection.png Photovoltaic Battery Charger - NiMh My initial data on the battery cell voltage drop while charging my iPhone. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1548023980996-26IASXT7VGDMNH7YX35Y/battery_charge_detector.png Photovoltaic Battery Charger - NiMh The 1N5817-T Schottky Diode prevents the batteries from charging the panel, while also only allowing the batteries charge to a max 11.55V. The 1N5348 zener diode only allows current to pass if the batteries have reached at least 11V. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1538952581636-V4YS95V9384W7J3QLENI/IMG_6633.jpg Photovoltaic Battery Charger - NiMh https://www.jackogrady.me/battery-management-system/overdischarge-protection daily 0.75 2021-07-19 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1555446333478-8JKAZUD036Z7EA98CR6O/voltage_regulators.jpg Overdischarge Protection - Overdischarge Protection Last Updated: 3/6/19 Li-ion cells must also be prevented from discharging below 2.5V, at which point the cells can attain irreversible damage that reduces their capacity. The voltage regulators in the photovoltaic battery charger effectively create an overdischarge protection mechanism. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1551936788065-5D6DHD6KCOMEPLO5YVSX/discharge_v4.png Overdischarge Protection - Copy of Copy of Copy of Copy of Battery Discharge Circuit The batteries are connected to two 5V regulators in parallel, which effectively prevent the cells from discharging past 5V (or 2.5V per cell). https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1551936449220-OJWZUGCJF91NXWUZ4787/voltage_regulators.jpg Overdischarge Protection - Copy of Copy of Copy of Copy of The Voltage Regulator Components A picture of the two voltage regulators. The power resistor and LED are not in the actual circuit and were placed for testing purposes. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1538952581636-V4YS95V9384W7J3QLENI/IMG_6633.jpg Overdischarge Protection https://www.jackogrady.me/battery-management-system/overcharge-protection daily 0.75 2021-07-19 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1553755567013-3CUYTLHZK8UVYL9MD9OZ/IMG_2205+%282%29.jpg Overcharge Protection - Overcharge Protection Last Updated: 3/6/19 To prevent the cells from charging past 4.2V/cell and achieving a dangerous energy level, I designed an overcharge protection circuit. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1551932765855-BCKPSHM1O101SQCF655X/IMG_2205+%281%29.jpg Overcharge Protection - Copy of Copy of Copy of A successful test of the overcharge protection circuit I am still working on increasing the current going to the cells, but this was a successful test run of the v7 circuit. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1551932382372-KAUY90S5Z5QYYR88U0HV/battery_overcharge_protection_v7.png Overcharge Protection - Copy of Copy of Copy of Overcharge Protection v7 This was my 7th version of the overcharge protection circuit, which I extensively tested in the lab using a power supply to mimic the battery cells. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1551932428237-095H38HU8GAVGS3ZT0RI/Vtotal_200+Ohm.png Overcharge Protection - Copy of Copy of Copy of Tweaking an earlier version of the circuit Current behavior near the desired cutoff voltage of 8.4V (in the v3 circuit). I discovered that a 200 Ohm resistor in series with the zener creates a premature reverse bias, https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1551932400376-A5KOZILWEFLGXHYG4MLY/overcharge_protection_v4.png Overcharge Protection - Copy of Copy of Copy of Overcharge Protection v4 This was an earlier version of the overcharge protection circuit which ultimately evolved into the v7 pictured earlier. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1538952581636-V4YS95V9384W7J3QLENI/IMG_6633.jpg Overcharge Protection https://www.jackogrady.me/battery-management-system daily 0.75 2021-07-19 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1555451306540-BWP1U2D82TWILUPBVQLN/full_system%40150x.png BMS Overview The photovoltaic battery charger the BMS is being designed for In this system, a 12V solar panel will charge 2 18650 Li-ion cells, which can then be used to charge an iPhone with sunlight. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1555449822473-3ZK4LTPMFHL9NFF8D99P/bms_cover_photo.jpg BMS Overview - Designing a Li-ion Battery Management System (BMS) Last updated: 5/13/19 In the process of designing a photovoltaic battery charger, I realized that I needed a BMS to safely operate and monitor the 18650 Li-ion cells that would be storing the solar energy. So, I decided to design my own. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1538952581636-V4YS95V9384W7J3QLENI/IMG_6633.jpg BMS Overview https://www.jackogrady.me/battery-management-system/state-of-charge daily 0.75 2022-09-14 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557792523291-ET8JA4O1TOZQSY9NN44F/adjusted_variance_matrices.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557789996353-9ZS40C7AWMKOYWOUISF5/repeat_var.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1556241383817-EALDT2XG5O2M48FL7EWS/image-asset.png State of Charge Estimation A matrix view of the SoC(OCV) function https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557788656633-LCO6QN39UMSEK9KJ0TE5/state_prediction.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557789393501-2WVBUPCK9PUGUIUXCP3J/residual_vector.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1555445661108-9EOZUBKV4Y3HY5E6SQ5N/kalman_filter_lab.jpg State of Charge Estimation - State of Charge Estimation with a Kalman Filter Last Updated: 5/13/19 State of charge (SoC) estimation is vitally important for practical battery applications. Here, I use a Kalman Filter running on Python and Arduino to estimate the SoC of the batteries. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557789652470-YIUQRZ0B106EZN5IZ6L1/new_var.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557786352801-1FZJKCBC1ZOEZ7XP66AW/labeled_system.png State of Charge Estimation Picture of the System in Operation All systems are labeled according to the diagram above. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557786718555-YJ1CUQGZHG9KQTMM3BXP/relay_on.png State of Charge Estimation Relay On: OCV Measurement Arduino writer D2 to HIGH, which energizes the relay and disconnects the batteries from load. Arduino measures the voltage drop between A1 and GND to determine the OCV of the batteries. R1 and R2 create a voltage divider so the OCV reading stays within the Arduino’s 5V measurement bounds. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1555821420996-8VU2P0BKD3IM6OJ5UMZI/ocv_collection.jpg State of Charge Estimation A Picture of the Open Circuit Voltage Measurement Circuit The Arduino is connected via USB to my MacBook, which runs a Python script that reads the OCV values from the Arduino and writes them to a .csv file https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557789048915-4R5OMQG3VA2A7DTRBQNQ/observation_matrix.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557786701774-4CBNC2IKIKCMTZPCKCPU/relay_off.png State of Charge Estimation Relay Off: Current Measurement The battery is connected to load and discharges through R-Power. Arduino measures the voltage drop across R-Power by measuring the drop between A0 and GND. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557787562242-Z6V8031MQSSYA5267UYG/KF_initialization.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1555808478310-VT1Q5EM2ZNTN49LW44XS/town_road%40150x.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557789402087-QXEJPJO2UQJTLC632E15/residual_variance.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557791531639-VMQXBU7AYKFYFSKZ5NKU/7hour_variance_matrices.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557789984120-00ESQPAE04YCQQHKQCCL/repeat_state_est.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557789036406-3WBSRYDHCTM1QGV7EY2L/state_meas.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1556241848664-8FO1YNLLZTEY7ANY0G1H/current_measurement_circuit.png State of Charge Estimation The Current Measurement Circuit Schematic An Arduino measures the voltage drop across R-Ammeter, then uses Ohm’s Law to calculate a current and sends it to Python via Serial. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557788666444-UBDCFNFV5M365Q71D6IN/essential_eqn.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557791949541-GOD6OSSAKEZEBNEWZ7Z9/overnight_comparison_difference.png State of Charge Estimation Kalman Filter vs Other Methods - Difference from “True” State of Charge The Kalman Filter’s estimate is nearly twice as accurate as pure coulomb counting. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1556240118673-KP332BHKF14LDXZJPU4J/image-asset.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1556242146395-2O90G8VZOM0CG73MNT2M/soc_used_calculation.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557788650589-UQYNACRE3E7IZDWV9F7Q/transformation_update.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1555818175049-H8ZNBF77SLN3NJEOJLS6/speedometer_method%40150x.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557794039137-7GB9STOC1TA6NKR0X38M/linear_nonlinear_regions.png State of Charge Estimation The Linear and Nonlinear Regions of a Li-ion Discharge Between 100% and 40% SoC, the Li-ion cell behaves fairly linearly; however, below 40% SoC, the cell begins to exhibit nonlinear behavior. This reduces the Kalman Filter’s accuracy below 40% SoC. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1555818187177-NVJ2W7WXU360Z9X07VKY/gps_method%40150x.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1555821385872-8Y9EMZ4N0IZMNVR70VGX/ocv_collection_circuit.png State of Charge Estimation The Open Circuit Voltage Measurement Circuit Schematic An Arduino measures the voltage drop across the cells using an analog pin. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1556242057552-0AEAR2W6XFERQ2NMVST0/coulomb_calculation.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557789644865-PIKTPNER3SKR8K6IG3HY/new_est.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557789410826-AKWXGNY46MVCYI7RY6D8/kalman_gain.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1556239900766-WJ6Q0S0B6MURCQF2NEXY/guassian_multiplication.png State of Charge Estimation Visualization of multiplying two Gaussians together, from https://www.bzarg.com/p/how-a-kalman-filter-works-in-pictures/ https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557789021559-DX0M9Y3MO7LES1AWUBK4/image-asset.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1555818200514-QS3SVBHMZBKPHJYY8D22/kalman_filter_method%40150x.png State of Charge Estimation https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557786135079-SHEH068I4XGRM0DFZAVP/image-asset.png State of Charge Estimation System Diagram Python controls the Arduino via the serial monitor, and Arduino manipulates which mode of data collection the circuit is in. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557791934857-66U6OSTE6DUS0Y47QWN8/overnight_comparison_discharge.png State of Charge Estimation Kalman Filter vs Other Methods - State of Charge Estimate The Kalman Filter is more accurate than coulomb counting and beats pure OCV measurements for about half of the experiment. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1555822745717-HZ94IUJNTOP0GHEM6YF7/battery_ocv_fitted.png State of Charge Estimation An 8th Order Polynomial Fit of the Battery Quasi-OCV Discharge The polynomial fit allows me to get a SoC estimate for a given OCV measurement https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557792908926-QZLZSB1TX35FSP9T3T79/2_hour_alt_methods.png State of Charge Estimation The Effect of Current Measurement Frequency and Variance Matrix Adjustments on the Kalman Filter’s Accuracy Increasing the frequency of current measurements had a negligible effect on the Kalman Filter’s accuracy. Increasing the estimate variance and reducing the measurement variance also had a negligible effect. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1557792923386-ODZFYBVYI5IAROSPIRI7/2_hour_alt_methods_difference.png State of Charge Estimation The Effect of Current Measurement Frequency and Variance Matrix Adjustments on the Kalman Filter’s Accuracy - Difference from “True” State of Charge While all Kalman Filter estimates beat the pure coulomb counting estimate, there was little difference between their estimates—even with different current measurement frequencies and variances matrices. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1538952581636-V4YS95V9384W7J3QLENI/IMG_6633.jpg State of Charge Estimation https://www.jackogrady.me/reinforcement-learning-solar/technical-implementation daily 0.75 2022-04-28 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/645a2f82-5b4d-425e-8be3-19fba862e88a/rl_project_overview.png Technical Implementation - Reinforcement Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/ada4762f-2683-45e6-a91f-ac79ce9359c3/osi_standard_logo_0.png Technical Implementation - Reinforcement Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1645474136000-XXP9VSHMRIKQMD5FMDRG/env_values.png Technical Implementation - Reinforcement Learning https://www.jackogrady.me/reinforcement-learning-solar/the-problem-statement daily 0.75 2022-09-05 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/66cb7f6c-4143-471d-9966-9cec88069459/solar_rl_drawings.jpeg The Problem Statement - Make it stand out Problem Complexities: In this figure, we see all the different factors at play which affect the optimal position of the solar panel https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/88d31072-becd-4247-a994-6df2e93f15de/solar_rl_drawings.jpeg The Problem Statement - The Problem Statement This page introduces the problem we’re attempting to solve with a reinforcement learning agent, as well as broadly the types of problems that RL is well-suited to solve. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1645496956430-1ESM60BHK2J9CGLN8NBQ/solar_rl_drawings.jpeg The Problem Statement https://www.jackogrady.me/reinforcement-learning-solar/dual-axis-panel-design daily 0.75 2022-09-05 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/3f94aa81-90b5-4454-b570-84e85f1cf1f4/design_annotation-01.png Dual-Axis Panel Design - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1627255547704-91DRAFHACMKHM9DYHZK7/panel_v1_design.png Dual-Axis Panel Design - Make it stand out Assembled view of the dual-axis panel design https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1627254787258-SNZWKCY933DRRYRDEUF5/panel_v1_design.png Dual-Axis Panel Design - Dual-Axis Panel Design Last updated: 4/27/22 This page contains the mechanical and electrical schematics for the dual-axis panel controlled by the reinforcement learning agent. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1627256346141-4LHMZKVAQA4WY2WV0HDW/panel_v1_exploded_view_1.png Dual-Axis Panel Design - Make it stand out Exploded view of the dual-axis panel design https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/bd987f7e-fd1e-4f0f-8348-1a458af2b887/panel_approach_angle.jpg Dual-Axis Panel Design - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1627255939235-4GXS1UHPWTIATDLIF313/panel_v1_design.png Dual-Axis Panel Design https://www.jackogrady.me/reinforcement-learning-solar/system-design daily 0.75 2022-09-05 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1628991456889-EODA1CPF03HDZFF2IFA7/solar-rl-system-schematic-system+view.png System Design - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1628991799658-CCV026E64Y0IODTPX5UU/solar-rl-system-schematic-agent_firmware_interface.png System Design - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1628990523520-9NBIF2V8JCNDOB8C9P98/solar-rl-system-schematic-system-cover+view.png System Design - System Design Last updated: 8/14/21 What does RL actually look like from a systems perspective? This page covers the system design, flow of information, and circuit schematic. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1628992231092-LFL39Z92SBXFKINGCKZE/solar-rl-system-schematic-system-cover+view.png System Design https://www.jackogrady.me/reinforcement-learning-solar/parts-list daily 0.75 2022-09-05 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1632637506890-8LWXYSC0ABBTK06ZAPTS/unsplash-image-bukjsECgmeU.jpg Parts List - Complete Parts List Last Updated: 9/26/21 This page holds the complete list of parts needed to re-create this project yourself. It’s currently a work in progress! https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1644299181513-EZ04RTNN4SOWC6ZINH8Q/unsplash-image-bukjsECgmeU.jpg Parts List https://www.jackogrady.me/reinforcement-learning-solar/circuit-schematic daily 0.75 2022-09-05 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1632699644385-1DV5COVVYS50AJJWYVB4/lab_bench_dual_axis.JPG Circuit Schematic - Circuit Schematic Last updated: 9/26/21 What circuit components are needed to control the dual-axis panel and measure power? This page covers the circuit schematic in detail! https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1632699703879-0AVI578MN6HO060ED3HS/solar-rl-system-schematic-circuit-sept-update.png Circuit Schematic - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1632701803311-DY30UGALGBHDWJIHORMG/lab_bench_comments.jpg Circuit Schematic - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1632702232626-NMQHB0CZV04UZLVPYNIG/lab_bench_dual_axis.JPG Circuit Schematic https://www.jackogrady.me/reinforcement-learning-solar/experiment-design daily 0.75 2022-09-05 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/fb1b293a-b653-4bf3-9447-8c129caefccb/scan_by_window.png Experiment Design - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/5782e732-2b27-4814-9c87-717cbdeb239b/heatmap_window_scan.png Experiment Design - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/ec9c2e05-a1e6-4b15-b4be-cac95b8a7d95/rl_experiment_lamp.JPG Experiment Design - Experiment Design This page details the generation of data for a simulated environment, and how we can assess agent performance based on this data https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/cb6fc83f-054c-4215-8aab-f32e5c963841/rl_data_generation.jpg Experiment Design - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/2cf6e896-b9ff-4a11-bbb0-64b9ec51bc24/rl_experiment_lamp.JPG Experiment Design - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/05a547aa-2c75-4710-b7ad-122c80b814cd/lamp_lighting+_situation.png Experiment Design - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/ea092a02-b8fe-47d8-9f54-6aafe847a238/performance_example.png Experiment Design - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/313a59e3-cb78-49dd-87d5-e2849629cf6f/env_values_scale.png Experiment Design - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1645310033399-I6S29OHYQ93P1CR3YL6P/scan_by_window.png Experiment Design https://www.jackogrady.me/misc/helpful-rl-resources daily 0.75 2022-09-11 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1644300457805-GZZ0RRNGOCIWFZUL1EM4/unsplash-image-lUaaKCUANVI.jpg Helpful RL Resources - Helpful RL Resources Online courses, books, and blog posts that I found helpful while getting into RL https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1644300383548-30XQEB13XPL8P2GBHC1I/unsplash-image-lUaaKCUANVI.jpg Helpful RL Resources https://www.jackogrady.me/reinforcement-learning-solar/rl-agent-softmax-actor-critic daily 0.75 2022-09-05 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/fd76ba68-74ab-44c7-92f9-372e27c8b15e/eqn_reward_function.png RL Agent - Softmax Actor-Critic - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1e7414b0-1a27-4329-92bb-13138b869ace/actor_critic_update.png RL Agent - Softmax Actor-Critic - Make it stand out Agent Updates: The error calculated updates probabilities of selecting actions in the actor, whereas it updates value estimates for states or actions in the critic. Action selection lives separately from action values. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/895888cd-085e-491e-b495-b2b119d1d471/policy_update_original.png RL Agent - Softmax Actor-Critic - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/61f121af-bd01-405c-837e-78c5c0351437/action_representation.png RL Agent - Softmax Actor-Critic - Make it stand out Value Representation: Both the critic and actor will utilize a table with row —> active state and columns —> actions in that state https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/037355a6-6021-4f75-b915-c8127346a3f7/action_selection_process.png RL Agent - Softmax Actor-Critic - Make it stand out Actor Policy: The diagram tries to visualize how the softmax probability influences the likelihood of actions being selected https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/9b9420db-f4db-4350-b4b6-ee68f041b7d0/gradient_simplification.png RL Agent - Softmax Actor-Critic - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/af89a48b-835c-49fc-9feb-3b1f05f8da4c/policy_gradient.png RL Agent - Softmax Actor-Critic - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/0b04a1da-16ea-4932-8585-c41d244601dc/state_action_interfaces.png RL Agent - Softmax Actor-Critic - Make it stand out Agent-System Interaction: The Arduino communicates with motor degree positions, and the agent utilizes a flat-map index. This diagram shows the conversion processes to go between the two. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/8a6786e4-e2a2-4add-8114-7c703e4bcc24/stacked_distributions.jpg RL Agent - Softmax Actor-Critic - Softmax Actor-Critic Implementation This page covers the technical implementation details of a softmax actor-critic agent to solve our energy optimization problem https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/7615ec68-c3e6-42d4-8ce1-eadd2264ab11/avg_reward_update_eqn.png RL Agent - Softmax Actor-Critic - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/34f8aa91-3b1f-41f3-bead-734d6f5ebd98/critic_update_eqn.png RL Agent - Softmax Actor-Critic - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/32dd7f58-5901-448c-8eab-f4ce0463932e/actor_critic_action_selection.png RL Agent - Softmax Actor-Critic - Make it stand out Action Selection Process: The critic is uninvolved with action selection https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/a08c2f3c-8f6e-4112-9de8-bdb18f66a918/actor_critic_overview.png RL Agent - Softmax Actor-Critic - Make it stand out Actor-Critic Overview: The process above loops continuously as the agent operates. Generally, the critic has a larger step size than the actor. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/64f38e83-57d0-49d1-9a95-d23a95ceeccb/delta_t_eqn.png RL Agent - Softmax Actor-Critic - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/94bdbdab-9ba9-4119-97e1-d9676e8ce099/agent_learning_flow.png RL Agent - Softmax Actor-Critic - Make it stand out Learning Process: An overview of the learning process for our agent, which repeats at each tilmestep from repeated agent-environment interactions. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/b27ced5c-ebdf-4a54-adde-45a0ab02d48e/eqn_motor_draw.png RL Agent - Softmax Actor-Critic - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/0ed28553-908a-4bd4-ab21-68389857c9e5/critic_network_update.png RL Agent - Softmax Actor-Critic - Make it stand out Critic State-Value Estimates: The critic considers the two states (current and previous) to calculate the value of being in each of these states, which is based on the q-values for all of the actions available from that state. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/e7d6b16b-4df6-495b-b573-c4f9d240cb4f/policy_update_final.png RL Agent - Softmax Actor-Critic - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1645310320361-WKVWGUGMDREAPZ0C3UPX/stacked_distributions.jpg RL Agent - Softmax Actor-Critic https://www.jackogrady.me/reinforcement-learning-solar/rl-agent-q-learning daily 0.75 2022-09-05 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/65306a99-f403-48ff-ae6a-787db86410c6/expected_sarsa_update.png RL Agent - Q-Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/168d3d21-ad2b-4517-9f60-eaa63d9aa8bc/q_learning_trace_optimal.png RL Agent - Q-Learning - Make it stand out Greedy Agent: In each of the three states, the agent happens to select the greedy action (which it does with probability 1 - epsilon) https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/972f5531-8ae9-45f4-aa49-041e66998f7e/performance_over_explore.png RL Agent - Q-Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/8a89f578-909d-4d60-85a5-0d2e195aa7cd/q_learning_cliff.png RL Agent - Q-Learning - Make it stand out Image from Adrian Yijie Xu on https://medium.com/gradientcrescent/fundamentals-of-reinforcement-learning-navigating-cliffworld-with-sarsa-and-q-learning-cc3c36eb5830 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/9320dee0-879b-4c06-95c8-7c7792711733/performance_small_epsilon.png RL Agent - Q-Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/130c2aa1-2c00-4bad-894e-dc3dc0e77a36/performance_smaller_step_size.png RL Agent - Q-Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/863e9a45-93b0-4c36-899d-aeed826c26d1/q_learning_trace_bad.png RL Agent - Q-Learning - Make it stand out Suboptimal Exploration: In state 2, the agent has selected a non-greedy action. It has epsilon probability of exploring, and then 1/N probability of selecting each action — regardless of their expected value (where N is the number of actions - 1) https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/2dd6f13d-202e-4ae1-b334-3c48d6bff073/q_learning_stacked_dist.jpg RL Agent - Q-Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/d5abf03a-a776-4ef8-b3d9-e9457fd1994b/performance_q_learning_1.png RL Agent - Q-Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/521b4918-2fc2-4573-a03a-ce47a3f03731/agent_state_visits.png RL Agent - Q-Learning - Make it stand out Figure 1: Number of times the agent had visited each state by step 24,000 in the experiment https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/057d9f1f-a9f7-4b56-923f-2607a63ddea5/q_learning_stacked_dist.jpg RL Agent - Q-Learning - Q-Learning Implementation This page covers the technical implementation of the original agent design, an episodic Q-learning agent with an epsilon-greedy policy https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/5d85ac57-c734-49fb-8412-efe2b57d4630/q_learning_update.png RL Agent - Q-Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/5e2910cd-b858-44a7-b5f7-417fab4f9f93/env_reward_step_compariosn.png RL Agent - Q-Learning - Make it stand out Figure 2: The environment reward for each state https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1645402061045-4ZVU2A4NK8KR38V5L8RX/q_learning_stacked_dist.jpg RL Agent - Q-Learning https://www.jackogrady.me/reinforcement-learning-solar/rl-fundamentals daily 0.75 2022-09-05 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/b2693e26-69c7-4dd7-a66d-406067f0b3dc/rl-algorithms-tree.png RL Fundamentals - Make it stand out You can click on the image above to enlarge https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/4ee4ca01-1173-49bf-a272-4b3d50f0bc15/rl_fundamentals_cover_3x.png RL Fundamentals - RL Fundamentals This page covers some key concepts around reinforcement learning, including Markov Decision Processes, agent components, and types of RL Agents. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/76209137-d568-42eb-8770-c30fef1911f6/agent_env_interaction_state_labels.png RL Fundamentals - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/07ee5a8c-bab6-488c-937d-b78ea511c56d/mdp_diagram.png RL Fundamentals - Make it stand out Example MDP: There are four states, and each state has multiple actions which transition to the next state. State transitions numbers are <probability of selecting action>, <reward for taking the action from the state>. For example, looking at the rightmost arrow: the agent in State 3 would select this action with a 33% chance and receive a +2 reward for taking such action from State 3 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/a03a9c96-5154-460d-8e21-b343096a4ea6/general_learning_process.png RL Fundamentals - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1645469328122-ARINO9CSWC87OHKXT4Y3/rl_fundamentals_cover_3x.png RL Fundamentals https://www.jackogrady.me/reinforcement-learning-solar/experiment-one-shifting-env daily 0.75 2022-09-05 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/00d9296a-9351-4451-a695-2d1804782977/scan_with_peak.png Experiment I: Shifting Env - Experiment I: Shifting Env This page covers the first full experiment using the softmax actor-critic agent https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/f70bb0e2-134c-459b-bfd6-c226fae0e7b0/softmax_static_performance.png Experiment I: Shifting Env - Make it stand out For context, the epsilon-greedy Q-learning agent would take closer to 20k steps to approach optimum in this experiment. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/6b97ff2c-e9f4-4113-8b84-a3aa67423714/heatmap_shifting.png Experiment I: Shifting Env - Make it stand out 2. Reward array for environment https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/0896ea67-02a9-447c-a91d-a6ee60c0f093/agent_performance.png Experiment I: Shifting Env - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/204a33ca-6c54-40c8-ac3f-7ac92b366476/scan_with_peak.png Experiment I: Shifting Env - Make it stand out 1. Collecting the light scan https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/d10bbea3-aaa8-48ce-bb08-dc7fb4c1ec10/critic_sensitivity.png Experiment I: Shifting Env - Make it stand out Example from more abbreviated hyperparameter study sweeping from 1e-4 to 1 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/53aca23c-7fe6-46b3-af1f-23b090444731/delta_flat.png Experiment I: Shifting Env - Make it stand out Delta (orange) flattens out https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1649087376045-5W22N9INIUO4Y10EBPDH/scan_with_peak.png Experiment I: Shifting Env https://www.jackogrady.me/reinforcement-learning-solar/experiment-two-shifting-env daily 0.75 2022-09-05 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1b3632f7-e1a8-4ce3-8a55-38873e1abae4/shifting_env_first_1m.png Experiment II: Shifting Env with Time of Day - Make it stand out First 1M steps of agent performance. The agent slowly converges too the optimal state and has some variability as it continues to optimize its behavior. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/3ff579f8-8504-4a45-9da4-5eba5626ef75/new_index_representation.png Experiment II: Shifting Env with Time of Day - Experiment II: Shifting Env with Time of Day This page covers an improved implementation of the softmax actor-critic where time of day is included as a state feature https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/204a33ca-6c54-40c8-ac3f-7ac92b366476/scan_with_peak.png Experiment II: Shifting Env with Time of Day - Make it stand out 1. Collecting the light scan https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/eb54b4a8-1d7d-4a16-85b1-aa280b2a1def/shifting_env_3m.png Experiment II: Shifting Env with Time of Day - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/6b97ff2c-e9f4-4113-8b84-a3aa67423714/heatmap_shifting.png Experiment II: Shifting Env with Time of Day - Make it stand out 2. Reward array for environment https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/638946bf-11fd-434b-83b4-c2db170822ea/shifting_env_last_1m.png Experiment II: Shifting Env with Time of Day - Make it stand out Last 1M steps of agent performance. The agent is consistently at or very near the optimal state value. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/7632fbac-0c5d-408a-bd30-979bd548b30a/time_of_day_index_addition-01.png Experiment II: Shifting Env with Time of Day - Make it stand out Visual representation of the initial array for actor and critic values as compared to the updated array structure, which is now three dimensional and has an index for time of day https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1649614708937-2S0BXXPD3KIULSEBCLS0/new_index_representation.png Experiment II: Shifting Env with Time of Day https://www.jackogrady.me/reinforcement-learning-solar/research-summary daily 0.75 2022-09-11 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/c047f77c-90e5-43cf-82fb-a936ba681542/delta_t_eqn.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out Delta equation for our agent https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/ba82eb1d-ea0c-46d7-beb6-77494fbe89ac/light_scan_process.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/3444cb96-445c-441e-882e-1404e8f161ee/traditional_development.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/66914c01-fdc9-4e79-96b7-19164e00099c/agent_indoor_performance.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/61739ba0-eb14-4254-95d7-8c277a4509cf/eqn_reward_function.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/11b36bfd-b50e-4339-aa8b-dae6329a05a3/action_selection_process.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out A visualization of how the actor’s learned values of each action influence its probability of being selected in a softmax policy https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/4c6b87f3-9aca-4487-a537-c9b7059412c0/critic_update_eqn.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out Critic update equation for our agent https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/f1e73e6f-7c4e-4035-9272-7c9ae960d0c4/heatmap_shifting.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out The 2D reward array based on the light scan. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/901301d2-4380-4443-b4ed-a0b719c16d69/agent-env-interaction.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out The agent interacts with the environment by requesting motor index positions, then learns from the power produced at those indices at that time of day https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/e83d8edc-b3b6-4fae-bd30-670ca8109a90/new_index_representation.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out A visualization of the three-dimensional structure of the actor and critic networks https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/9db5246a-470b-4966-8d5b-32377c8d27ee/agent_outdoor_last_1M.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/9cd7723c-6442-4142-9bc2-1d42ba1778ad/agent_outdoor_first_1M.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/05256fff-4d42-4513-bcdb-31517ac9b39d/rl_development_process.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/ee257ab1-27bf-491f-b42c-726619522c70/3d_light_scan.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out The light scan from the video above in 3D https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/651bb053-556e-4429-9a6c-0561ddeb1fc6/scan_with_peak.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out A separate light scan with more pronounced peaks and troughs. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/bda6723d-99f0-4186-b180-b0b86640f030/general_learning_process.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/03441c90-5bd4-42b1-8f8b-71c0ffeedb1b/rl_solar_cover_image_4_3.png Optimizing Solar Energy Production with Reinforcement Learning - Optimizing Solar Energy Production with Reinforcement Learning https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/996fe1b1-0223-4595-987f-7306ca21eb6c/eqn_motor_draw.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out Sigma is defined as some constant. In this implementation, sigma is set to 0.0001W per degree of movement per motor https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/fc74a0f6-f128-46be-9093-d748576ce368/design_annotation-01-01.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out Detailed view of the dual-axis panel hardware https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1650828636908-Z2QTG862UTDPN90GURXM/panel_v1_design.png Optimizing Solar Energy Production with Reinforcement Learning https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1650828654262-TXH4A1JD01T9TYTQNEPW/panel_approach_angle.jpg Optimizing Solar Energy Production with Reinforcement Learning https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/f32dc349-4a05-41d6-82c9-bbf3e0c16c9a/agent_outdoor_performance.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/04ccad3e-91e7-4273-9f09-abc8d1217b48/policy_update_final.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out Actor update equation for our agent. See RL Agent - Softmax Actor-Critic for details on this gradient simplification. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/6475ef6b-43b3-4d80-9d58-084431036964/2d_light_scan.png Optimizing Solar Energy Production with Reinforcement Learning - Make it stand out The light scan from the video above as the 2D reward array passed to the environment class https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1651163413601-0G6URUK8RP4KHF4L95CY/new_aspect_image-04.png Optimizing Solar Energy Production with Reinforcement Learning https://www.jackogrady.me/misc/a-semitechnical-introduction-to-ai daily 0.75 2022-12-08 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/9dc65859-fa9a-4ffa-bc34-ff61e40fc61f/table_missing_values.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/84924e52-e537-44de-aa76-14bf2a3c3dd1/server_cooling.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/ea7c554e-fa47-40a8-b26b-6d8d2957f462/alpha_go.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/44af338d-52ef-4a9b-8ce2-9e3f83115e45/wooden_surface_simple.jpg A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/84bac87c-7a5c-48cf-b3b6-c89820b003f5/dalle_2.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/34548ea0-9eef-47fd-afbc-9316c1c47cf4/domain_table_2.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/04315e66-cbcf-470a-8398-de47c59aaf10/woodent_surface_problem.jpg A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/92b5fa60-3401-4838-b89b-48a2c28fd716/weather_price_neural_network.png A Semi-Technical Introduction to AI - Make it stand out Neural network representation of the price-prediction problem. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/590ceb66-2311-4edc-b10b-8d6064d016ed/alphafold_post.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/a2dc5c0c-9d08-4fef-9e3b-dc8b8c306493/nuclear_fusion.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/63a4605a-b856-4f7d-b1cc-5dea6d56e646/domain_table_3.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/02a3b5b5-2c47-4652-8262-7fd0a98005f6/DALL%C2%B7E+2022-07-17+16.19.04+-+Artificial+intelligence+by+Salvador+Dali.png A Semi-Technical Introduction to AI - A Semi-Technical Introduction to AI The image is what DALLE-2 (an AI art generator) created for my prompt: “Artificial Intelligence by Salvador Dali” https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/92b22a0b-5ddd-4deb-94e0-fae1d6f29ce2/neural_network_flow.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/baad5423-8a06-43b6-b80d-aecd9940cde6/gt_sophy.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/2ea8b406-e2c5-4f6d-baff-14efd2be483b/neural_net_tensors.png A Semi-Technical Introduction to AI - Make it stand out Technically speaking, there’s also some addition to a bias tensor at each layer, as well as a special activation function, but that’s outside the scope of this article. We’re just peaking the interest and planting some seeds here. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/746fc429-81ca-48b6-826c-a524cb694008/tesla_self_driving.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/14c08c0c-6db1-44a2-aa11-fe88808fcc4c/simple_rl_initial_values.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/633223ef-86c1-4497-9a0e-05a5572c4828/extra_column.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/510809de-2c61-44c5-a0a9-40c23aaa216f/model_regression.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/e83ae45e-0593-4a6f-b5ba-6fe7741bd747/double_rows.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/3edd8dbf-9c6f-4a47-a67c-12193ea2047a/ai_wind_prediction.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/9666b1c8-286c-4efd-ba1f-dadf7b9f174c/rl_complex_table.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/4aa31389-fad7-4b3e-a110-5fd87ef7df87/initial_table.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/d1619edd-0b52-4a7b-8bd5-28d9301e6f11/jack_solar_panel_rl.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/a7dade66-b099-4655-8fba-4835bf9effad/simple_rl_learned_values.png A Semi-Technical Introduction to AI - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/a01439b4-04a3-41f8-ba64-f710b2d67950/deep_rl_network.png A Semi-Technical Introduction to AI - Make it stand out A hypothetical neural network used to approximate the action-values of our RL agent https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/1662865262708-XD7PFNJ0XEGTB2Q35MG9/DALL%25C2%25B7E%2B2022-07-17%2B16.19.04%2B-%2BArtificial%2Bintelligence%2Bby%2BSalvador%2BDali.jpg A Semi-Technical Introduction to AI https://www.jackogrady.me/about-me daily 1.0 2022-09-14 https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/adf77db2-bcc7-401d-a090-1ecffa14e446/DALL%C2%B7E+2022-09-13+12.56.07+-+wind+farms+as+single+line+art.jpg About Me - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/fd6f0ad9-ff69-4036-9d85-21c9b4431920/surf_yew_lighter.jpg About Me - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/fe612803-4533-4990-9fed-41e8c4eb2f27/DALL%C2%B7E+2022-09-11+23.47.08+-+a+picture+of+a+solar+farm+shot+on+a+disposable+camera.png About Me - Make it stand out Whatever it is, the way you tell your story online can make all the difference. https://images.squarespace-cdn.com/content/v1/5bba8415c2ff6147bdcf9a89/0120615e-750b-4265-a9be-060f16ed3df9/r1t_lighter.jpg About Me - Make it stand out Whatever it is, the way you tell your story online can make all the difference.