{"group":{"id":1,"name":"Community","lockable":false,"created_at":"2012-01-18T18:02:15.000Z","updated_at":"2025-12-14T01:33:56.000Z","description":"Problems submitted by members of the MATLAB Central community.","is_default":true,"created_by":161519,"badge_id":null,"featured":false,"trending":false,"solution_count_in_trending_period":0,"trending_last_calculated":"2025-12-14T00:00:00.000Z","image_id":null,"published":true,"community_created":false,"status_id":2,"is_default_group_for_player":false,"deleted_by":null,"deleted_at":null,"restored_by":null,"restored_at":null,"description_opc":null,"description_html":null,"published_at":null},"problems":[{"id":1983,"title":"Big data","description":"Optimize this line of code:\r\n\r\n B = sum(gradient(corrcoef(A)).^2);\r\n \r\nfor a matrix A with size(A,2)\u003e\u003esize(A,1)\r\n\r\n*Description:*\r\n\r\nAnalyses of large datasets often require some level of optimization for speed and/or memory usage. This is an example problem where both the initial data A and final measure of interest B fit perfectly well in memory, but the intermediate variables (in this case an impossibly large correlation matrix) required to compute the final measure of interest do not. \r\n\r\nWe have a large 2-dimensional matrix A (with dimensions 100 x 100,000).\r\n\r\nWe need to compute the following row vector B (with dimensions 1 x 100,000):\r\n\r\n B = sum(gradient(corrcoef(A)).^2);\r\n \r\nThis computes first the matrix of correlation coefficients for each pair of columns in A:\r\n\r\n a = corrcoef(A)\r\n\r\n(a 100,000 by 100,000 matrix), then computes the spatial derivative of the resulting correlation matrix along the second dimension:\r\n\r\n b = gradient(a)\r\n\r\n(another 100,000 by 100,000 matrix), and finally computes the squared norm of each column in the resulting matrix:\r\n\r\n B = sum(b.^2,1) \r\n\r\n(a 100,000 element vector)\r\n\r\nThis straight-forward \"vectorized\" approach, nevertheless, fails because it requires too much memory (enough to store a 100,000 x 100,000 correlation matrix, around 80Gb). \r\n\r\nWe clearly need some form of simplification/optimization. Can you compute the measure B within the time-limit of a Cody solution? (typically about 30 seconds)\r\n\r\nSolutions will be scored based on computation time (score equal to total time in seconds). \r\n\r\n*Context:* (not relevant to solving this problem)\r\n\r\nThis problem arises in the analyses of fMRI datasets. A typical result from a fMRI scanner session is a 4-dimensional matrix A(x,y,z,t), where the first three dimensions are spatial dimensions (a scanner of the subject's head/brain) and the fourth dimension is temporal (sequential scans obtained during a typical fMRI session). Think of these as time-varying three-dimensional pictures of your brain activation. A lot of research in the past few years has focused on functional connectivity, a measure of the temporal correlation between the \"activation\" of any pair of brain areas. Several recent papers have investigated the possibility to obtain entirely data-driven parcellations of the brain (partitioning the brain into functionally-homogeneous areas) based on these spatial patterns of functional connectivity. The measure _B_ above represents one of the measures that have been suggested as a way to drive these data-driven parcellations (borders between two adjacent but functionally distinct brain areas are expected to show higher spatial gradients in functional connectivity profiles). For simplicity I have collapsed the three spatial dimensions into one for this problem, but the computational complexity of the original computation is approximately the same (a typical scanner session results in something of the order of several hundred thousands \"voxels\" -three dimensional \"pixels\"- within the brain, and a few hundred time-points; this makes computing the entire \"voxel-to-voxel\" correlation matrix, or measures derived from it, rather challenging).\r\n\r\n","description_html":"\u003cp\u003eOptimize this line of code:\u003c/p\u003e\u003cpre\u003e B = sum(gradient(corrcoef(A)).^2);\u003c/pre\u003e\u003cp\u003efor a matrix A with size(A,2)\u0026gt;\u0026gt;size(A,1)\u003c/p\u003e\u003cp\u003e\u003cb\u003eDescription:\u003c/b\u003e\u003c/p\u003e\u003cp\u003eAnalyses of large datasets often require some level of optimization for speed and/or memory usage. This is an example problem where both the initial data A and final measure of interest B fit perfectly well in memory, but the intermediate variables (in this case an impossibly large correlation matrix) required to compute the final measure of interest do not.\u003c/p\u003e\u003cp\u003eWe have a large 2-dimensional matrix A (with dimensions 100 x 100,000).\u003c/p\u003e\u003cp\u003eWe need to compute the following row vector B (with dimensions 1 x 100,000):\u003c/p\u003e\u003cpre\u003e B = sum(gradient(corrcoef(A)).^2);\u003c/pre\u003e\u003cp\u003eThis computes first the matrix of correlation coefficients for each pair of columns in A:\u003c/p\u003e\u003cpre\u003e a = corrcoef(A)\u003c/pre\u003e\u003cp\u003e(a 100,000 by 100,000 matrix), then computes the spatial derivative of the resulting correlation matrix along the second dimension:\u003c/p\u003e\u003cpre\u003e b = gradient(a)\u003c/pre\u003e\u003cp\u003e(another 100,000 by 100,000 matrix), and finally computes the squared norm of each column in the resulting matrix:\u003c/p\u003e\u003cpre\u003e B = sum(b.^2,1) \u003c/pre\u003e\u003cp\u003e(a 100,000 element vector)\u003c/p\u003e\u003cp\u003eThis straight-forward \"vectorized\" approach, nevertheless, fails because it requires too much memory (enough to store a 100,000 x 100,000 correlation matrix, around 80Gb).\u003c/p\u003e\u003cp\u003eWe clearly need some form of simplification/optimization. Can you compute the measure B within the time-limit of a Cody solution? (typically about 30 seconds)\u003c/p\u003e\u003cp\u003eSolutions will be scored based on computation time (score equal to total time in seconds).\u003c/p\u003e\u003cp\u003e\u003cb\u003eContext:\u003c/b\u003e (not relevant to solving this problem)\u003c/p\u003e\u003cp\u003eThis problem arises in the analyses of fMRI datasets. A typical result from a fMRI scanner session is a 4-dimensional matrix A(x,y,z,t), where the first three dimensions are spatial dimensions (a scanner of the subject's head/brain) and the fourth dimension is temporal (sequential scans obtained during a typical fMRI session). Think of these as time-varying three-dimensional pictures of your brain activation. A lot of research in the past few years has focused on functional connectivity, a measure of the temporal correlation between the \"activation\" of any pair of brain areas. Several recent papers have investigated the possibility to obtain entirely data-driven parcellations of the brain (partitioning the brain into functionally-homogeneous areas) based on these spatial patterns of functional connectivity. The measure \u003ci\u003eB\u003c/i\u003e above represents one of the measures that have been suggested as a way to drive these data-driven parcellations (borders between two adjacent but functionally distinct brain areas are expected to show higher spatial gradients in functional connectivity profiles). For simplicity I have collapsed the three spatial dimensions into one for this problem, but the computational complexity of the original computation is approximately the same (a typical scanner session results in something of the order of several hundred thousands \"voxels\" -three dimensional \"pixels\"- within the brain, and a few hundred time-points; this makes computing the entire \"voxel-to-voxel\" correlation matrix, or measures derived from it, rather challenging).\u003c/p\u003e","function_template":"function B = bigdata(A)\r\nB=zeros(1,size(A,2));\r\nend","test_suite":"%%\r\nglobal score time_count;\r\nscore=60; time_count=[];\r\nassessFunctionAbsence({'global'},'FileName','bigdata.m');\r\n\r\n%%\r\n% test small matrices first\r\nA=randn(100,100);\r\nB=bigdata(A);\r\nB0=sum(gradient(corrcoef(A)).^2);\r\nassert(all(abs(B-B0)\u003c1e-4));\r\n\r\n%%\r\n% test small matrices first\r\nA=randn(100,1000);\r\nt0=clock;\r\nB=bigdata(A);\r\nt1=etime(clock,t0);\r\nB0=sum(gradient(corrcoef(A)).^2);\r\nassert(all(abs(B-B0)\u003c1e-4));\r\nfprintf('Test 1 (1000 columns). Ellapsed time %f\\n',t1);\r\n\r\n%%\r\n% test small matrices first\r\nglobal time_count;\r\nA=randn(100,2000);\r\nt0=clock;\r\nB=bigdata(A);\r\nt1=etime(clock,t0);\r\nB0=sum(gradient(corrcoef(A)).^2);\r\nassert(all(abs(B-B0)\u003c1e-4));\r\nfprintf('Test 2 (2000 columns). Ellapsed time %f\\n',t1);\r\ntime_count=t1;\r\n\r\n%%\r\n% test small matrices first\r\nglobal time_count;\r\nA=randn(100,3000);\r\nt0=clock;\r\nB=bigdata(A);\r\nt1=etime(clock,t0);\r\nB0=sum(gradient(corrcoef(A)).^2);\r\nassert(all(abs(B-B0)\u003c1e-4));\r\nfprintf('Test 3 (3000 columns). Ellapsed time %f\\n',t1);\r\ntime_count=[time_count t1];\r\n\r\n%%\r\n% test small matrices first\r\nglobal time_count;\r\nA=randn(100,4000);\r\nt0=clock;\r\nB=bigdata(A);\r\nt1=etime(clock,t0);\r\nB0=sum(gradient(corrcoef(A)).^2);\r\nassert(all(abs(B-B0)\u003c1e-4));\r\nfprintf('Test 4 (4000 columns). Ellapsed time %f\\n',t1);\r\ntime_count=[time_count t1];\r\n\r\n%%\r\n% test big matrix\r\nglobal score time_count;\r\nassert(sum(time_count)\u003c2,'small matrices take too long; skipping big matrix computation to avoid time-out');\r\n\r\nrandn('seed',0);\r\nA=randn(100,100000);\r\nt0=clock;\r\nB=bigdata(A);\r\nt1=etime(clock,t0);\r\n\r\nassert(all(abs(B([81473 90580 12699 91338 63236 9755 27850 54689 95751 96489 15762 97060 95717 48538 80029 14189 42177 91574 79221 95950 65575 3572 84913 93400 67874])-[446.969229611013 529.483918542325 492.884459351883 506.453343056463 557.687700571202 570.008636747334 546.678181864512 445.139644002495 459.617263841976 565.276303202775 481.70081228927 505.718997924023 515.061623552965 363.542792068668 482.02142593325 578.480557740742 553.42526785998 470.314702013725 604.606838616675 539.148055906616 450.528222494512 491.803528433601 547.232718164205 561.773622227699 520.453638191039])\u003c1e-4));\r\nfprintf('Test 5 (100000 columns). Ellapsed time %f\\n',t1);\r\ntime_count=[time_count t1];\r\nscore=min(60,round(sum(time_count)));\r\n\r\n%%\r\nglobal score;\r\nurlwrite('https://sites.google.com/a/alfnie.com/alfnie/software/SetSolutionScore.p?attredirects=0\u0026amp;d=1','SetSolutionScore.p');\r\nrehash path;\r\nSetSolutionScore(score);\r\n\r\n","published":true,"deleted":false,"likes_count":2,"comments_count":3,"created_by":43,"edited_by":null,"edited_at":null,"deleted_by":null,"deleted_at":null,"solvers_count":4,"test_suite_updated_at":"2017-10-27T00:44:32.000Z","rescore_all_solutions":true,"group_id":1,"created_at":"2013-11-12T05:45:56.000Z","updated_at":"2017-10-27T00:44:32.000Z","published_at":"2013-11-13T03:31:33.000Z","restored_at":null,"restored_by":null,"spam":false,"simulink":false,"admin_reviewed":false,"description_opc":"{\"relationships\":[{\"relationshipType\":\"http://schemas.mathworks.com/matlab/code/2013/relationships/document\",\"targetMode\":\"\",\"relationshipId\":\"rId1\",\"target\":\"/matlab/document.xml\"},{\"relationshipType\":\"http://schemas.mathworks.com/matlab/code/2013/relationships/output\",\"targetMode\":\"\",\"relationshipId\":\"rId2\",\"target\":\"/matlab/output.xml\"}],\"parts\":[{\"partUri\":\"/matlab/document.xml\",\"relationship\":[],\"contentType\":\"application/vnd.mathworks.matlab.code.document+xml\",\"content\":\"\u003c?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?\u003e\\n\u003cw:document xmlns:w=\\\"http://schemas.openxmlformats.org/wordprocessingml/2006/main\\\"\u003e\u003cw:body\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eOptimize this line of code:\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"code\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e\u003c![CDATA[ B = sum(gradient(corrcoef(A)).^2);]]\u003e\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003efor a matrix A with size(A,2)\u0026gt;\u0026gt;size(A,1)\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:rPr\u003e\u003cw:b/\u003e\u003c/w:rPr\u003e\u003cw:t\u003eDescription:\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eAnalyses of large datasets often require some level of optimization for speed and/or memory usage. This is an example problem where both the initial data A and final measure of interest B fit perfectly well in memory, but the intermediate variables (in this case an impossibly large correlation matrix) required to compute the final measure of interest do not.\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eWe have a large 2-dimensional matrix A (with dimensions 100 x 100,000).\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eWe need to compute the following row vector B (with dimensions 1 x 100,000):\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"code\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e\u003c![CDATA[ B = sum(gradient(corrcoef(A)).^2);]]\u003e\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eThis computes first the matrix of correlation coefficients for each pair of columns in A:\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"code\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e\u003c![CDATA[ a = corrcoef(A)]]\u003e\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e(a 100,000 by 100,000 matrix), then computes the spatial derivative of the resulting correlation matrix along the second dimension:\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"code\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e\u003c![CDATA[ b = gradient(a)]]\u003e\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e(another 100,000 by 100,000 matrix), and finally computes the squared norm of each column in the resulting matrix:\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"code\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e\u003c![CDATA[ B = sum(b.^2,1)]]\u003e\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e(a 100,000 element vector)\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eThis straight-forward \\\"vectorized\\\" approach, nevertheless, fails because it requires too much memory (enough to store a 100,000 x 100,000 correlation matrix, around 80Gb).\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eWe clearly need some form of simplification/optimization. Can you compute the measure B within the time-limit of a Cody solution? (typically about 30 seconds)\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eSolutions will be scored based on computation time (score equal to total time in seconds).\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:rPr\u003e\u003cw:b/\u003e\u003c/w:rPr\u003e\u003cw:t\u003eContext:\u003c/w:t\u003e\u003c/w:r\u003e\u003cw:r\u003e\u003cw:t\u003e (not relevant to solving this problem)\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eThis problem arises in the analyses of fMRI datasets. A typical result from a fMRI scanner session is a 4-dimensional matrix A(x,y,z,t), where the first three dimensions are spatial dimensions (a scanner of the subject's head/brain) and the fourth dimension is temporal (sequential scans obtained during a typical fMRI session). Think of these as time-varying three-dimensional pictures of your brain activation. A lot of research in the past few years has focused on functional connectivity, a measure of the temporal correlation between the \\\"activation\\\" of any pair of brain areas. Several recent papers have investigated the possibility to obtain entirely data-driven parcellations of the brain (partitioning the brain into functionally-homogeneous areas) based on these spatial patterns of functional connectivity. The measure\u003c/w:t\u003e\u003c/w:r\u003e\u003cw:r\u003e\u003cw:t\u003e \u003c/w:t\u003e\u003c/w:r\u003e\u003cw:r\u003e\u003cw:rPr\u003e\u003cw:i/\u003e\u003c/w:rPr\u003e\u003cw:t\u003eB\u003c/w:t\u003e\u003c/w:r\u003e\u003cw:r\u003e\u003cw:t\u003e above represents one of the measures that have been suggested as a way to drive these data-driven parcellations (borders between two adjacent but functionally distinct brain areas are expected to show higher spatial gradients in functional connectivity profiles). For simplicity I have collapsed the three spatial dimensions into one for this problem, but the computational complexity of the original computation is approximately the same (a typical scanner session results in something of the order of several hundred thousands \\\"voxels\\\" -three dimensional \\\"pixels\\\"- within the brain, and a few hundred time-points; this makes computing the entire \\\"voxel-to-voxel\\\" correlation matrix, or measures derived from it, rather challenging).\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003c/w:body\u003e\u003c/w:document\u003e\"},{\"partUri\":\"/matlab/output.xml\",\"contentType\":\"text/xml\",\"content\":\"\u003c?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\" standalone=\\\"no\\\" ?\u003e\u003cembeddedOutputs\u003e\u003cmetaData\u003e\u003cevaluationState\u003emanual\u003c/evaluationState\u003e\u003clayoutState\u003ecode\u003c/layoutState\u003e\u003coutputStatus\u003eready\u003c/outputStatus\u003e\u003c/metaData\u003e\u003coutputArray type=\\\"array\\\"/\u003e\u003cregionArray type=\\\"array\\\"/\u003e\u003c/embeddedOutputs\u003e\"}]}"}],"problem_search":{"errors":[],"problems":[{"id":1983,"title":"Big data","description":"Optimize this line of code:\r\n\r\n B = sum(gradient(corrcoef(A)).^2);\r\n \r\nfor a matrix A with size(A,2)\u003e\u003esize(A,1)\r\n\r\n*Description:*\r\n\r\nAnalyses of large datasets often require some level of optimization for speed and/or memory usage. This is an example problem where both the initial data A and final measure of interest B fit perfectly well in memory, but the intermediate variables (in this case an impossibly large correlation matrix) required to compute the final measure of interest do not. \r\n\r\nWe have a large 2-dimensional matrix A (with dimensions 100 x 100,000).\r\n\r\nWe need to compute the following row vector B (with dimensions 1 x 100,000):\r\n\r\n B = sum(gradient(corrcoef(A)).^2);\r\n \r\nThis computes first the matrix of correlation coefficients for each pair of columns in A:\r\n\r\n a = corrcoef(A)\r\n\r\n(a 100,000 by 100,000 matrix), then computes the spatial derivative of the resulting correlation matrix along the second dimension:\r\n\r\n b = gradient(a)\r\n\r\n(another 100,000 by 100,000 matrix), and finally computes the squared norm of each column in the resulting matrix:\r\n\r\n B = sum(b.^2,1) \r\n\r\n(a 100,000 element vector)\r\n\r\nThis straight-forward \"vectorized\" approach, nevertheless, fails because it requires too much memory (enough to store a 100,000 x 100,000 correlation matrix, around 80Gb). \r\n\r\nWe clearly need some form of simplification/optimization. Can you compute the measure B within the time-limit of a Cody solution? (typically about 30 seconds)\r\n\r\nSolutions will be scored based on computation time (score equal to total time in seconds). \r\n\r\n*Context:* (not relevant to solving this problem)\r\n\r\nThis problem arises in the analyses of fMRI datasets. A typical result from a fMRI scanner session is a 4-dimensional matrix A(x,y,z,t), where the first three dimensions are spatial dimensions (a scanner of the subject's head/brain) and the fourth dimension is temporal (sequential scans obtained during a typical fMRI session). Think of these as time-varying three-dimensional pictures of your brain activation. A lot of research in the past few years has focused on functional connectivity, a measure of the temporal correlation between the \"activation\" of any pair of brain areas. Several recent papers have investigated the possibility to obtain entirely data-driven parcellations of the brain (partitioning the brain into functionally-homogeneous areas) based on these spatial patterns of functional connectivity. The measure _B_ above represents one of the measures that have been suggested as a way to drive these data-driven parcellations (borders between two adjacent but functionally distinct brain areas are expected to show higher spatial gradients in functional connectivity profiles). For simplicity I have collapsed the three spatial dimensions into one for this problem, but the computational complexity of the original computation is approximately the same (a typical scanner session results in something of the order of several hundred thousands \"voxels\" -three dimensional \"pixels\"- within the brain, and a few hundred time-points; this makes computing the entire \"voxel-to-voxel\" correlation matrix, or measures derived from it, rather challenging).\r\n\r\n","description_html":"\u003cp\u003eOptimize this line of code:\u003c/p\u003e\u003cpre\u003e B = sum(gradient(corrcoef(A)).^2);\u003c/pre\u003e\u003cp\u003efor a matrix A with size(A,2)\u0026gt;\u0026gt;size(A,1)\u003c/p\u003e\u003cp\u003e\u003cb\u003eDescription:\u003c/b\u003e\u003c/p\u003e\u003cp\u003eAnalyses of large datasets often require some level of optimization for speed and/or memory usage. This is an example problem where both the initial data A and final measure of interest B fit perfectly well in memory, but the intermediate variables (in this case an impossibly large correlation matrix) required to compute the final measure of interest do not.\u003c/p\u003e\u003cp\u003eWe have a large 2-dimensional matrix A (with dimensions 100 x 100,000).\u003c/p\u003e\u003cp\u003eWe need to compute the following row vector B (with dimensions 1 x 100,000):\u003c/p\u003e\u003cpre\u003e B = sum(gradient(corrcoef(A)).^2);\u003c/pre\u003e\u003cp\u003eThis computes first the matrix of correlation coefficients for each pair of columns in A:\u003c/p\u003e\u003cpre\u003e a = corrcoef(A)\u003c/pre\u003e\u003cp\u003e(a 100,000 by 100,000 matrix), then computes the spatial derivative of the resulting correlation matrix along the second dimension:\u003c/p\u003e\u003cpre\u003e b = gradient(a)\u003c/pre\u003e\u003cp\u003e(another 100,000 by 100,000 matrix), and finally computes the squared norm of each column in the resulting matrix:\u003c/p\u003e\u003cpre\u003e B = sum(b.^2,1) \u003c/pre\u003e\u003cp\u003e(a 100,000 element vector)\u003c/p\u003e\u003cp\u003eThis straight-forward \"vectorized\" approach, nevertheless, fails because it requires too much memory (enough to store a 100,000 x 100,000 correlation matrix, around 80Gb).\u003c/p\u003e\u003cp\u003eWe clearly need some form of simplification/optimization. Can you compute the measure B within the time-limit of a Cody solution? (typically about 30 seconds)\u003c/p\u003e\u003cp\u003eSolutions will be scored based on computation time (score equal to total time in seconds).\u003c/p\u003e\u003cp\u003e\u003cb\u003eContext:\u003c/b\u003e (not relevant to solving this problem)\u003c/p\u003e\u003cp\u003eThis problem arises in the analyses of fMRI datasets. A typical result from a fMRI scanner session is a 4-dimensional matrix A(x,y,z,t), where the first three dimensions are spatial dimensions (a scanner of the subject's head/brain) and the fourth dimension is temporal (sequential scans obtained during a typical fMRI session). Think of these as time-varying three-dimensional pictures of your brain activation. A lot of research in the past few years has focused on functional connectivity, a measure of the temporal correlation between the \"activation\" of any pair of brain areas. Several recent papers have investigated the possibility to obtain entirely data-driven parcellations of the brain (partitioning the brain into functionally-homogeneous areas) based on these spatial patterns of functional connectivity. The measure \u003ci\u003eB\u003c/i\u003e above represents one of the measures that have been suggested as a way to drive these data-driven parcellations (borders between two adjacent but functionally distinct brain areas are expected to show higher spatial gradients in functional connectivity profiles). For simplicity I have collapsed the three spatial dimensions into one for this problem, but the computational complexity of the original computation is approximately the same (a typical scanner session results in something of the order of several hundred thousands \"voxels\" -three dimensional \"pixels\"- within the brain, and a few hundred time-points; this makes computing the entire \"voxel-to-voxel\" correlation matrix, or measures derived from it, rather challenging).\u003c/p\u003e","function_template":"function B = bigdata(A)\r\nB=zeros(1,size(A,2));\r\nend","test_suite":"%%\r\nglobal score time_count;\r\nscore=60; time_count=[];\r\nassessFunctionAbsence({'global'},'FileName','bigdata.m');\r\n\r\n%%\r\n% test small matrices first\r\nA=randn(100,100);\r\nB=bigdata(A);\r\nB0=sum(gradient(corrcoef(A)).^2);\r\nassert(all(abs(B-B0)\u003c1e-4));\r\n\r\n%%\r\n% test small matrices first\r\nA=randn(100,1000);\r\nt0=clock;\r\nB=bigdata(A);\r\nt1=etime(clock,t0);\r\nB0=sum(gradient(corrcoef(A)).^2);\r\nassert(all(abs(B-B0)\u003c1e-4));\r\nfprintf('Test 1 (1000 columns). Ellapsed time %f\\n',t1);\r\n\r\n%%\r\n% test small matrices first\r\nglobal time_count;\r\nA=randn(100,2000);\r\nt0=clock;\r\nB=bigdata(A);\r\nt1=etime(clock,t0);\r\nB0=sum(gradient(corrcoef(A)).^2);\r\nassert(all(abs(B-B0)\u003c1e-4));\r\nfprintf('Test 2 (2000 columns). Ellapsed time %f\\n',t1);\r\ntime_count=t1;\r\n\r\n%%\r\n% test small matrices first\r\nglobal time_count;\r\nA=randn(100,3000);\r\nt0=clock;\r\nB=bigdata(A);\r\nt1=etime(clock,t0);\r\nB0=sum(gradient(corrcoef(A)).^2);\r\nassert(all(abs(B-B0)\u003c1e-4));\r\nfprintf('Test 3 (3000 columns). Ellapsed time %f\\n',t1);\r\ntime_count=[time_count t1];\r\n\r\n%%\r\n% test small matrices first\r\nglobal time_count;\r\nA=randn(100,4000);\r\nt0=clock;\r\nB=bigdata(A);\r\nt1=etime(clock,t0);\r\nB0=sum(gradient(corrcoef(A)).^2);\r\nassert(all(abs(B-B0)\u003c1e-4));\r\nfprintf('Test 4 (4000 columns). Ellapsed time %f\\n',t1);\r\ntime_count=[time_count t1];\r\n\r\n%%\r\n% test big matrix\r\nglobal score time_count;\r\nassert(sum(time_count)\u003c2,'small matrices take too long; skipping big matrix computation to avoid time-out');\r\n\r\nrandn('seed',0);\r\nA=randn(100,100000);\r\nt0=clock;\r\nB=bigdata(A);\r\nt1=etime(clock,t0);\r\n\r\nassert(all(abs(B([81473 90580 12699 91338 63236 9755 27850 54689 95751 96489 15762 97060 95717 48538 80029 14189 42177 91574 79221 95950 65575 3572 84913 93400 67874])-[446.969229611013 529.483918542325 492.884459351883 506.453343056463 557.687700571202 570.008636747334 546.678181864512 445.139644002495 459.617263841976 565.276303202775 481.70081228927 505.718997924023 515.061623552965 363.542792068668 482.02142593325 578.480557740742 553.42526785998 470.314702013725 604.606838616675 539.148055906616 450.528222494512 491.803528433601 547.232718164205 561.773622227699 520.453638191039])\u003c1e-4));\r\nfprintf('Test 5 (100000 columns). Ellapsed time %f\\n',t1);\r\ntime_count=[time_count t1];\r\nscore=min(60,round(sum(time_count)));\r\n\r\n%%\r\nglobal score;\r\nurlwrite('https://sites.google.com/a/alfnie.com/alfnie/software/SetSolutionScore.p?attredirects=0\u0026amp;d=1','SetSolutionScore.p');\r\nrehash path;\r\nSetSolutionScore(score);\r\n\r\n","published":true,"deleted":false,"likes_count":2,"comments_count":3,"created_by":43,"edited_by":null,"edited_at":null,"deleted_by":null,"deleted_at":null,"solvers_count":4,"test_suite_updated_at":"2017-10-27T00:44:32.000Z","rescore_all_solutions":true,"group_id":1,"created_at":"2013-11-12T05:45:56.000Z","updated_at":"2017-10-27T00:44:32.000Z","published_at":"2013-11-13T03:31:33.000Z","restored_at":null,"restored_by":null,"spam":false,"simulink":false,"admin_reviewed":false,"description_opc":"{\"relationships\":[{\"relationshipType\":\"http://schemas.mathworks.com/matlab/code/2013/relationships/document\",\"targetMode\":\"\",\"relationshipId\":\"rId1\",\"target\":\"/matlab/document.xml\"},{\"relationshipType\":\"http://schemas.mathworks.com/matlab/code/2013/relationships/output\",\"targetMode\":\"\",\"relationshipId\":\"rId2\",\"target\":\"/matlab/output.xml\"}],\"parts\":[{\"partUri\":\"/matlab/document.xml\",\"relationship\":[],\"contentType\":\"application/vnd.mathworks.matlab.code.document+xml\",\"content\":\"\u003c?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?\u003e\\n\u003cw:document xmlns:w=\\\"http://schemas.openxmlformats.org/wordprocessingml/2006/main\\\"\u003e\u003cw:body\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eOptimize this line of code:\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"code\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e\u003c![CDATA[ B = sum(gradient(corrcoef(A)).^2);]]\u003e\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003efor a matrix A with size(A,2)\u0026gt;\u0026gt;size(A,1)\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:rPr\u003e\u003cw:b/\u003e\u003c/w:rPr\u003e\u003cw:t\u003eDescription:\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eAnalyses of large datasets often require some level of optimization for speed and/or memory usage. This is an example problem where both the initial data A and final measure of interest B fit perfectly well in memory, but the intermediate variables (in this case an impossibly large correlation matrix) required to compute the final measure of interest do not.\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eWe have a large 2-dimensional matrix A (with dimensions 100 x 100,000).\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eWe need to compute the following row vector B (with dimensions 1 x 100,000):\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"code\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e\u003c![CDATA[ B = sum(gradient(corrcoef(A)).^2);]]\u003e\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eThis computes first the matrix of correlation coefficients for each pair of columns in A:\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"code\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e\u003c![CDATA[ a = corrcoef(A)]]\u003e\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e(a 100,000 by 100,000 matrix), then computes the spatial derivative of the resulting correlation matrix along the second dimension:\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"code\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e\u003c![CDATA[ b = gradient(a)]]\u003e\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e(another 100,000 by 100,000 matrix), and finally computes the squared norm of each column in the resulting matrix:\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"code\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e\u003c![CDATA[ B = sum(b.^2,1)]]\u003e\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003e(a 100,000 element vector)\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eThis straight-forward \\\"vectorized\\\" approach, nevertheless, fails because it requires too much memory (enough to store a 100,000 x 100,000 correlation matrix, around 80Gb).\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eWe clearly need some form of simplification/optimization. Can you compute the measure B within the time-limit of a Cody solution? (typically about 30 seconds)\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eSolutions will be scored based on computation time (score equal to total time in seconds).\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:rPr\u003e\u003cw:b/\u003e\u003c/w:rPr\u003e\u003cw:t\u003eContext:\u003c/w:t\u003e\u003c/w:r\u003e\u003cw:r\u003e\u003cw:t\u003e (not relevant to solving this problem)\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003cw:p\u003e\u003cw:pPr\u003e\u003cw:pStyle w:val=\\\"text\\\"/\u003e\u003c/w:pPr\u003e\u003cw:r\u003e\u003cw:t\u003eThis problem arises in the analyses of fMRI datasets. A typical result from a fMRI scanner session is a 4-dimensional matrix A(x,y,z,t), where the first three dimensions are spatial dimensions (a scanner of the subject's head/brain) and the fourth dimension is temporal (sequential scans obtained during a typical fMRI session). Think of these as time-varying three-dimensional pictures of your brain activation. A lot of research in the past few years has focused on functional connectivity, a measure of the temporal correlation between the \\\"activation\\\" of any pair of brain areas. Several recent papers have investigated the possibility to obtain entirely data-driven parcellations of the brain (partitioning the brain into functionally-homogeneous areas) based on these spatial patterns of functional connectivity. The measure\u003c/w:t\u003e\u003c/w:r\u003e\u003cw:r\u003e\u003cw:t\u003e \u003c/w:t\u003e\u003c/w:r\u003e\u003cw:r\u003e\u003cw:rPr\u003e\u003cw:i/\u003e\u003c/w:rPr\u003e\u003cw:t\u003eB\u003c/w:t\u003e\u003c/w:r\u003e\u003cw:r\u003e\u003cw:t\u003e above represents one of the measures that have been suggested as a way to drive these data-driven parcellations (borders between two adjacent but functionally distinct brain areas are expected to show higher spatial gradients in functional connectivity profiles). For simplicity I have collapsed the three spatial dimensions into one for this problem, but the computational complexity of the original computation is approximately the same (a typical scanner session results in something of the order of several hundred thousands \\\"voxels\\\" -three dimensional \\\"pixels\\\"- within the brain, and a few hundred time-points; this makes computing the entire \\\"voxel-to-voxel\\\" correlation matrix, or measures derived from it, rather challenging).\u003c/w:t\u003e\u003c/w:r\u003e\u003c/w:p\u003e\u003c/w:body\u003e\u003c/w:document\u003e\"},{\"partUri\":\"/matlab/output.xml\",\"contentType\":\"text/xml\",\"content\":\"\u003c?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\" standalone=\\\"no\\\" ?\u003e\u003cembeddedOutputs\u003e\u003cmetaData\u003e\u003cevaluationState\u003emanual\u003c/evaluationState\u003e\u003clayoutState\u003ecode\u003c/layoutState\u003e\u003coutputStatus\u003eready\u003c/outputStatus\u003e\u003c/metaData\u003e\u003coutputArray type=\\\"array\\\"/\u003e\u003cregionArray type=\\\"array\\\"/\u003e\u003c/embeddedOutputs\u003e\"}]}"}],"term":"tag:\"big data\"","current_player_id":null,"fields":[{"name":"page","type":"integer","callback":null,"default":1,"directive":null,"facet":null,"facet_method":"and","operator":null,"param":null,"static":null,"prepend":true},{"name":"per_page","type":"integer","callback":null,"default":50,"directive":null,"facet":null,"facet_method":"and","operator":null,"param":null,"static":null,"prepend":true},{"name":"sort","type":"string","callback":null,"default":null,"directive":null,"facet":null,"facet_method":"and","operator":null,"param":null,"static":null,"prepend":true},{"name":"body","type":"text","callback":null,"default":"*:*","directive":null,"facet":null,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":false},{"name":"group","type":"string","callback":null,"default":null,"directive":"group","facet":true,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":true},{"name":"difficulty_rating_bin","type":"string","callback":null,"default":null,"directive":"difficulty_rating_bin","facet":true,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":true},{"name":"id","type":"integer","callback":null,"default":null,"directive":"id","facet":null,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":true},{"name":"tag","type":"string","callback":null,"default":null,"directive":"tag","facet":null,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":true},{"name":"product","type":"string","callback":null,"default":null,"directive":"product","facet":null,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":true},{"name":"created_at","type":"timeframe","callback":{},"default":null,"directive":"created_at","facet":null,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":true},{"name":"profile_id","type":"integer","callback":null,"default":null,"directive":"author_id","facet":null,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":true},{"name":"created_by","type":"string","callback":null,"default":null,"directive":"author","facet":null,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":true},{"name":"player_id","type":"integer","callback":null,"default":null,"directive":"solver_id","facet":null,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":true},{"name":"player","type":"string","callback":null,"default":null,"directive":"solver","facet":null,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":true},{"name":"solvers_count","type":"integer","callback":null,"default":null,"directive":"solvers_count","facet":null,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":true},{"name":"comments_count","type":"integer","callback":null,"default":null,"directive":"comments_count","facet":null,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":true},{"name":"likes_count","type":"integer","callback":null,"default":null,"directive":"likes_count","facet":null,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":true},{"name":"leader_id","type":"integer","callback":null,"default":null,"directive":"leader_id","facet":null,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":true},{"name":"leading_solution","type":"integer","callback":null,"default":null,"directive":"leading_solution","facet":null,"facet_method":"and","operator":null,"param":"term","static":null,"prepend":true}],"filters":[{"name":"asset_type","type":"string","callback":null,"default":null,"directive":null,"facet":null,"facet_method":"and","operator":null,"param":null,"static":"\"cody:problem\"","prepend":true},{"name":"profile_id","type":"integer","callback":{},"default":null,"directive":null,"facet":null,"facet_method":"and","operator":null,"param":"author_id","static":null,"prepend":true}],"query":{"params":{"per_page":50,"term":"tag:\"big data\"","current_player":null,"sort":"map(difficulty_value,0,0,999) asc"},"parser":"MathWorks::Search::Solr::QueryParser","directives":{"term":{"directives":{"tag":[["tag:\"big data\"","","\"","big data","\""]]}}},"facets":{"#\u003cMathWorks::Search::Field:0x00007f4f4603afb0\u003e":null,"#\u003cMathWorks::Search::Field:0x00007f4f4603af10\u003e":null},"filters":{"#\u003cMathWorks::Search::Field:0x00007f4f46039f70\u003e":"\"cody:problem\""},"fields":{"#\u003cMathWorks::Search::Field:0x00007f4f4603b230\u003e":1,"#\u003cMathWorks::Search::Field:0x00007f4f4603b190\u003e":50,"#\u003cMathWorks::Search::Field:0x00007f4f4603b0f0\u003e":"map(difficulty_value,0,0,999) asc","#\u003cMathWorks::Search::Field:0x00007f4f4603b050\u003e":"tag:\"big data\""},"user_query":{"#\u003cMathWorks::Search::Field:0x00007f4f4603b050\u003e":"tag:\"big data\""},"queried_facets":{}},"query_backend":{"connection":{"configuration":{"index_url":"http://index-op-v2/solr/","query_url":"http://search-op-v2/solr/","direct_access_index_urls":["http://index-op-v2/solr/"],"direct_access_query_urls":["http://search-op-v2/solr/"],"timeout":10,"vhost":"search","exchange":"search.topic","heartbeat":30,"pre_index_mode":false,"host":"rabbitmq-eks","port":5672,"username":"search","password":"J3bGPZzQ7asjJcCk","virtual_host":"search","indexer":"amqp","http_logging":"true","core":"cody"},"query_connection":{"uri":"http://search-op-v2/solr/cody/","proxy":null,"connection":{"parallel_manager":null,"headers":{"User-Agent":"Faraday v1.0.1"},"params":{},"options":{"params_encoder":"Faraday::FlatParamsEncoder","proxy":null,"bind":null,"timeout":null,"open_timeout":null,"read_timeout":null,"write_timeout":null,"boundary":null,"oauth":null,"context":null,"on_data":null},"ssl":{"verify":true,"ca_file":null,"ca_path":null,"verify_mode":null,"cert_store":null,"client_cert":null,"client_key":null,"certificate":null,"private_key":null,"verify_depth":null,"version":null,"min_version":null,"max_version":null},"default_parallel_manager":null,"builder":{"adapter":{"name":"Faraday::Adapter::NetHttp","args":[],"block":null},"handlers":[{"name":"Faraday::Response::RaiseError","args":[],"block":null}],"app":{"app":{"ssl_cert_store":{"verify_callback":null,"error":null,"error_string":null,"chain":null,"time":null},"app":{},"connection_options":{},"config_block":null}}},"url_prefix":"http://search-op-v2/solr/cody/","manual_proxy":false,"proxy":null},"update_format":"RSolr::JSON::Generator","update_path":"update","options":{"url":"http://search-op-v2/solr/cody"}}},"query":{"params":{"per_page":50,"term":"tag:\"big data\"","current_player":null,"sort":"map(difficulty_value,0,0,999) asc"},"parser":"MathWorks::Search::Solr::QueryParser","directives":{"term":{"directives":{"tag":[["tag:\"big data\"","","\"","big data","\""]]}}},"facets":{"#\u003cMathWorks::Search::Field:0x00007f4f4603afb0\u003e":null,"#\u003cMathWorks::Search::Field:0x00007f4f4603af10\u003e":null},"filters":{"#\u003cMathWorks::Search::Field:0x00007f4f46039f70\u003e":"\"cody:problem\""},"fields":{"#\u003cMathWorks::Search::Field:0x00007f4f4603b230\u003e":1,"#\u003cMathWorks::Search::Field:0x00007f4f4603b190\u003e":50,"#\u003cMathWorks::Search::Field:0x00007f4f4603b0f0\u003e":"map(difficulty_value,0,0,999) asc","#\u003cMathWorks::Search::Field:0x00007f4f4603b050\u003e":"tag:\"big data\""},"user_query":{"#\u003cMathWorks::Search::Field:0x00007f4f4603b050\u003e":"tag:\"big data\""},"queried_facets":{}},"options":{"fields":["id","difficulty_rating"]},"join":" "},"results":[{"id":1983,"difficulty_rating":"unrated"}]}}