This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/LCA.2015.2458318, IEEE Computer Architecture Letters

A Coarse-Grained Reconfigurable Architecture for

Compute-Intensive MapReduce Acceleration
Shuang Liang, Shouyi Yin, Leibo Liu, Yike Guo, and Shaojun Wei,

AbstractLarge-scale workloads often show parallelism of different levels. which offers acceleration potential for clusters and parallel
processors. Although processors such as GPGPUs and FPGAs show good performance of speedup, there is still vacancy for a low
power, high efficiency and dynamically reconfigurable one, and coarse-grained reconfigurable architecture(CGRA) seems to be one
possible choice. In this paper, we introduce how we use our CGRA fabric Chameleon to realize a dynamically reconfigurable
acceleration to MapReduce-based(MR-based) applications. A FPGA-shell-CGRA-core(FSCC) architecture is designed for the
acceleration PCI-Express board, and a programming model with compilation flow for CGRA is presented. With the supports above, a
small evaluation cluster with Hadoop framework is set up, and experiments on compute-intensive applications show that the
programming process is significantly simplified, with an 30-60x speedup offered under low power.

Index TermsReconfigurable computing, MapReduce, Accelerator

F
1 I NTRODUCTION

A S the performance requirement of computing rapidly

grows, constructing a more powerful engine for data
processing becomes a necessity. Accumulating computing cores
and raising clock frequency seems to be one solution, and
spreading workloads among a scalable multi-node cluster can
break the single-node limitation of storage and throughput.
However, the physical and financial constraints are preventing
us from the former approach. The root cause is that from the
aspect of processing element level, the Von-Neumann based
general-purpose processors(GPPs) are of low efficiency, which
lead to serious energy consumption and throughput gap for
their serial working style. Thus, it is desirable to take advan-
tages of more efficient and flexible processors.
GPGPUs [13] [6] have been widely used for distributed
acceleration due to their abundant stream processing units and
convenient programming models. They bring appealing speed
together with a headache of watt number. FPGAs [16] [15] [14]
[10] have also been developed recently to enhance the perfor-
mance of large-scale tasks, which show 10-1000x through-put
than GPPs with less than 10 watts power on a single chip, prov-
ing the high efficiency of its flexible and parallel work style.
Yet we should see that it is really time consuming to realize a
certain function on FPGA with bothersome RTL programming,
even if researchers have paid great efforts on implementing
tools to generate RTL codes from high-level languages [12] [11].
Moreover, it takes minutes to hours to generate a feasible bit-
stream for FPGAs due to the high interconnection complexity,
and hence makes the dynamic reconfiguration impractical.
The coarse-grained reconfigurable architectures(CGRAs)
seem to be suitable substitutes, which consist of a proper num-
ber of coarse-grained processing elements(PEs), typically word-
level ALUs instead of bit-level LUTs, as depicted in Figure 1.
The complexity of interconnections, usually crossbar or mesh
styled, is significantly reduced after the shrink of PE number,
which results in less synthesis time aiming for the on-the-fly
Fig. 1. Typical architecture of CGRA
programmability, and less power consumption and latencies on
unnecessary interconnects for higher on-chip efficiency. [8] has
offered fundamental support that coarse-grained architectures
will show averagely 10 times speed and a quarter energy cost
of LUT-based FPGAs.
Acknowledgedly, MapReduce [4] has been a well-suited
programming model for processing most large-scale work-
loads in distributed clusters. It provides two primitive functions
map and reduce to split the workloads into < key, value >
pairs for parallel executions, and what is more, among those
compute-intensive workloads such as matrix multiplication, pi
calculation and k-means clustering, etc., the overall patterns
of the algorithms are often made of repetitive simple kernels
in a loop form. This offers opportunity for CGRAs to explore
the parallelism of iterations within a limited hardware space,
which leads us to think of introducing CGRAs into the world
of MapReduce for a mutually rein-force.
In this paper, we are going to describe how we integrate
our Chameleon CGRA in an MR-based system. Hardware and
software implementations will be narrated in detail, and some
case studies will show the effectiveness of CGRA acceleration.

S.Liang, S.Yin, L.Liu and S.Wei are with the Institute of Microelectronics,
Tsinghua University, Beijing, China, 100084. S.Yin is the corresponding
author.
E-mail: yinsy@tsinghua.edu.cn
Y.Guo is with the Department of Computing, Imperial College London,
UK.
2 CGRA- CORED ACCELERATOR BOARD DESIGN
2.1 Architecture and working mechanism of our CGRA
Based on the typical CGRA architecture shown in Figure 1,
we implemented a prototype chip called Chameleon with
TSMC 65nm LP1P8M CMOS technology, which has an area of

1556-6056 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/LCA.2015.2458318, IEEE Computer Architecture Letters

Fig. 2. Hierarchical structure of configuration context storage

6.2 6.2mm2 and a maximum clock of 400MHz. The power

consumption of the chip is 49.8mW under 200MHz.
Each chip consists of four 8 8 2-D mesh PE arrays and
configuration context(CC) memories. Instead of FPGAs bit-
streams, the functional pattern of PEs are expressed by word-
wide(32bit) CCs, which can be easily translated from high-
level instructions. CCs are organized in a three configuration-
levels(CLs) hierarchical structure depicted in Figure 2. We
represent the kernels in the form of controlled data flow graph-
s(CDFGs), which can be mapped onto the PE array(PEA) under
the guidance of CCs. Each CLs responsibility is listed below:
CL2: The bottom level CC. The 5-bit opcode is used to
configure the function of ALUs, which supports 26 different
kinds of arithmetic and logic operators including add, subtract,
multiply, and, not, etc.. I/O configuration will direct the inter-
connections between two adjacent PEs and the dependence of
iterations, thus forming a kernel pattern and temporal schedule.
CL1: The medium level CC. It includes the index of CL2, and
indicates the information of iterations, such as the number of
iterations and the dependence among iterations.
CL0: The top level CC. It includes the index of CL1, and
offers the addresses of external data operations. The target PEA
number and synchronization control info are also given in CL0.
This makes our Chameleon works in a different way from
FPGA. For FPGAs, all the logics should be carefully designed
first under the resource limitations, then programmed with RTL
codes, and finally a lengthy run of synthesis and P&R, which is
a long development cycle. Admittedly this offers plenty of flexi-
bility for custom design, but it is too much for describing simple
algorithms. For our CGRA, the workloads are abstracted in a
higher level with several key parameters: iteration times, kernel
patterns, input and output address and interconnections. As a
result, the complexity of describing a workload is significantly
reduced. Designers only need to define parameters without too
much worry about the bottom hardware.
We also notice that between two adjacent iterations, the
kernel pattern may need to be reconfigured recurrently, so we
store CL1 and CL2 in the on-chip context memories to avoid
redundant communication, and only needs to write CL0 words
to push the overall workflow forward in a streaming mode.
2.2 Board design
Practically, we need a high speed bus to transfer data in
two-way and ensure that the bandwidth will not become a
bottleneck. PCI-Express bus is a suitable choice for it can bring a
peak full-duplex data transfer speed of 4GByte/s, so we choose
to integrate Chameleon on a board with PCI-Express interface.
Furthermore, peripheral logics are required to communicate
and cooperate CGRAs with the host server. For communication,
A PCI-Express endpoint IP should be set to collect and transmit
transaction level packages(TLPs) from and to the server. A
direct memory access(DMA) module is necessary for fast data
transferring between the host servers memory and RAM on
board. A data cache memory and the according interface logic
(a)
(b)
Fig. 3. Implementation of the accelerator board: (a)The FSCC structure
design. (b)The photo of the prototype PCI-express board integrated with
two Chameleon CGRAs
is also needed. For cooperation control, an interrupt controller
can catch the start and the end of one certain task, and a system
controller is in charge of the overall on-board system control.
Control and status registers can tell the functional modules to
work as arranged, and offer feedbacks for their inside status
machine and the server monitoring.
All the above modules need little reconfiguration at run-
time. They can be statically implemented with RTL and
mapped onto hardware such as FPGA. Therefore, we couple
a FPGA chip with CGRAs to realize a FPGA-shell-CGRA-
core(FSCC) structure, where the computing kernels are mapped
on CGRAs with the controlling of static logic on FPGA. Con-
sidering the logic cell and I/O requirement, we select a Virtex-6
LX550T chip together with two Chameleon chips on board. Two
Samsung K7N643645M 8MByte pipelined SRAMs are utilized
as a on-board cache storage for the data transferred between
the host server and CGRAs. We use a synchronous clock of
200MHz for the system, and realize a 3.27Gbytes/s of PCI-E
communication speed. The FSCC structure in our system is
shown in Figure 3(a), and the photo of the prototype PCI-E
board is given in Figure 3(b).
3 MR- BASED PROGRAMMING MODEL FOR CGRA
MapReduce offers an approach for developers to simply
write map and reduce functions to enable a distributed execution
of applications. We need to fit the task into the MapReduce
model first. Here we take matrix multiplication as an example,
and the pseudo-code is given in Algorithm 1. In this example,
the elements in the source matrices are replicated and labeled
with the key values, which are the target locations, in the map

1556-6056 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/LCA.2015.2458318, IEEE Computer Architecture Letters

Algorithm 1 Mapper for matrix multiplication

1: INPUT: (key,value) //value is (A, i, j, aij ) or (B, j, k, bjk )
2: if value[0]==A then
3: i=value[1];j=value[2];aij =value[3];
4: for k=1 to p do
5: emit((i, k), (A, j, aij ));
6: end for
7: else
8: j=value[1];k=value[2];bjk =value[3]; Fig. 4. The compilation flow of CIPs in MapReduce functions
9: for i=1 to m do
10: emit((i, k), (B, j, ajk ));
11: end for
12: end if

Algorithm 2 Reducer for matrix multiplication

1: INPUT: (key,value)
2: //key is(i,k), values is a list of (A, j, aij ) or (B, j, bjk )
3: hashA = {j : aij f or(x, j, aij ) in values if x == A}
4: hashB = {j : bjk f or(x, j, bjk ) in values if x == B}
5: result = 0
6: for j=1 to n do
7: result+ = hashA [j] hashB [j]; Fig. 5. The block diagram of the prototype Hadoop-based cluster with
8: end for our accelerator boards inserted
9: emit(key,result);

function. The reduce function searches for the elements with the
same key value, and yields vector inner productions in parallel.
After the map and reduce functions presented, we should
consider how to compile the compute-intensive parts of the
functions to offload them onto CGRAs. Just as shown in Figure
4, the compute-intensive parts(CIPs) can always be present-
ed as control data flow graphs(CDFGs). With the hardware
parameters such as PE number and memory bandwidth, the
original CDFG will be optimized by transformations [5] to
suit with Chameleon array. Then the original CDFG will be
transformed into a series of subgraphs, and the key parameters
such as kernel pattern, subgraph iteration number, iteration
dependence and input data address will be abstracted. With
these parameters, configuration contexts will be generated and
packed up in a parametric execution package. The above pro-
cedures can be called as a compilation flow for Chameleon,
which can be accomplished manually or by a custom compiler,
and currently we have developed an LLVM-based compiler [9]
[3], which can go through the procedures automatically.
We write a C driver for the access of bottom hardware due
to the register arrangement, and use Java native interface(JNI)
calling C libraries and linking it with MapReduce applications.
As shown in Figure 4, with the pointers of source and end
given, the exe CGRA function will realize the CIPs in the
original mapper/reducer on CGRAs.
4 E VALUATION SYSTEM
To verify the effectiveness CGRA acceleration, we need to set
up a distributed environment. Since Hadoop [2] is a widely
used framework of MapReduce, and it offers abundant tool-
s and libraries for development, so we choose to set up a
Hadoop-based cluster. The overall system workflow is given
in Figure 5. Here we apply five IBM x3650 M4 servers with
a Xeon E5-2650 2GHz CPU and 8GB ECC DDR3 memory in
each, connected with local network. In Hadoop framework,
Namenode is the manager of the whole system, and Datanodes
work under the control of Namenode with status feedbacks for
TABLE 1
MapReduce models for the benchmark applications
Function MM KMC CONV
Euclid distance,
Mapper Replicate Multiply
Compare, Sum
Reducer Multiply, Sum
Euclid distance,
Sum
Sum, Mean
Complexity O(M N P ) O(N KD) O(K 2 N 2 )
Tested size M=N=P=128 N=10e5,K=10,D=2 K=7,N=224
scheduling reference. In our cluster, four servers are configured
as Datanodes and each inserted with one Chameleon accelera-
tor boards through the PCI-Express slots.
5 E XPERIMENTAL RESULTS
Our experiments mainly focus on three aspects: the simplicity
of CGRA programming, the standalone performance improve-
ment on CGRAs and the overall improvement in the MR-based
prototype cluster. The benchmark applications we have chosen
are: matrix multiplication(MM), K-means clustering(KMC), and
2-D convolution(CONV) in convolutional neural networks, for
they all show compute-intensive characteristics and are easy to
be paralleled. The MapReduce models, the timing complexities
and the tested size of these applications are listed in Table 1.
We extract the CIPs in the MapReduce functions for acceler-
ation. To compare the strengths and weaknesses between FPGA
and CGRA implementations, the CIPs are realized in both
HDL and C++. We respectively generate configurations using
Xilinx ISE Design suite v14.7 for FPGA and our LLVM-based
C compiler for Chameleon. For the configuration generation
time(CGT), FPGA takes 10-14 minutes to generate bitstream
files with 90.3-112.1 Mb size, while CGRA only takes 23-37
seconds to generate execution packages with 1.65-5.15Kb size.
The main difference comes from the elimination of structure
description of PEs and the reduction of interconnection number
and combinations. Therefore, FPGAs can only be statically con-
figured before the chip actually works, while our Chameleon
CGRA can be dynamically configured with much simpler para-
metric description of the workload provided.

1556-6056 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/LCA.2015.2458318, IEEE Computer Architecture Letters

TABLE 2
Comparison between CPU, FPGA and CGRA
Xeon in 2GHz Virtex 6 LX550T in 200MHz Chameleon in 200MHz
Func # of operations
Latency(ms) Cycles Latency(ms) Speedup Cycles Latency(ms) Speedup
MM 4.19e6 14.41 59.96 16385 8.19e-2 175.89 4.59e3 77824 3.89e-1 37.03 1.09e5
KMC 8.10e6 12.29 135.77 41032 2.05e-1 59.90 3.49e3 53125 2.66e-1 46.27 3.08e5
CONV 4.74e6 20.07 48.68 47526 2.37e-1 84.46 1.77e3 72604 3.63e-1 55.29 1.32e5
*
Here we set the energy efficiency as = ( of operations)/(P ower(mW ) Execution time(ms))

ACKNOWLEDGMENTS
This work is supported by the National Nature Science founda-
tion of China(No.61274131), the International S&T Cooperation
Project of China(No. 2012DFA11170), the Tsinghua Indigenous
Research Project(No.20111080997) and the China National High
Technologies Research Program(No. 2012-AA012701).

(a) (b)
Fig. 6. The speedup of three applications with different number of nodes:
(a)non-accelerated; (b)CGRA-accelerated.
For the standalone performance, we implement three ver-
sions on CPU, FPGA and CGRA respectively. The detailed
comparison is given in Table 2. The power consumption of Xeon
CPU is measured by the Powerstat tool [1] with a wattsUp
PRO watt meter. As we can see, both FPGA and CGRA
shows significant speedup and power advantages comparing
with CPU. However, due to the limitation of PE number(512),
Chameleon shows worse speedup than Virtex-6 FPGA, while
offering a high efficiency which is almost 30-90 times of FPGAs.
We should mention that Chameleons technology is 65nm while
Virtex-6 is 40nm, and the area of Chameleon is only a quarter
of Virtex-6. Due to the scaling principle [7], with reduction of
CGRA I/O number, more CGRA PEs will be integrated on
board, which will bring an even better timing improvement.
Finally we test the applications in a CGRA accelerated
cluster environment. We rewrite the applications in MapReduce
forms, and make another version with the loops(the CIPs)
compiled into CGRA execution packages and called by the
exe CGRA function with the key parameters defined. We take
the non-accelerated single node case as the baseline, and we
test the applications with different Datanode number from one
to four. We process 104 copies of the applications, in order to
raise the datasets to a Gigabyte level. The normalized speedup
of the cluster is shown in Figure 6. We can observe that the
relationship between the node number and speedup is non-
linear, for the I/O communication ratio will raise relatively
as the single-node compute time drops. However, we can see
that the node number does not affect the effect of CGRA-
acceleration very much, for the PCI-Express bus brings fast
communication and the parallelism on CGRAs remains stable.
6 C ONCLUSION
In this paper, we present how we bring CGRA accelerator in
a MR-based system, with hardware architecture and software
programming model declared. Dynamic programmability has
been primarily achieved, and a considerable speedup and a re-
markable energy efficiency value is realized in both standalone
and cluster cases. The power saving will be even more tempting
if the cluster scale grows big enough. Future work lies in the
CGRA mapping optimization and a real-time scheduler with
CGRA status feedbacks.
R EFERENCES
[1] Powerstat for desktop. http://sourceforge.net/projects/
powerstatfordes/.
[2] D. Borthakur. The hadoop distributed file system: Architecture
and design. Hadoop Project Website, 11:21, 2007.
[3] Y. Chongyong, Y. Shouyi, L. Leibo, and W. Shaojun. Compiler
framework for reconfigurable computing architecture. IEICE trans-
actions on electronics, 92(10):12841290, 2009.
[4] J. Dean and S. Ghemawat. Mapreduce: simplified data processing
on large clusters. Communications of the ACM, 51(1):107113, 2008.
[5] V. Govindaraju, C.-H. Ho, T. Nowatzki, J. Chhugani, N. Satish,
K. Sankaralingam, and C. Kim. Dyser: Unifying functionality and
parallelism specialization for energy-efficient computing. IEEE
Micro, (5):3851, 2012.
[6] B. He, W. Fang, Q. Luo, N. K. Govindaraju, and T. Wang. Mars: a
mapreduce framework on graphics processors. In Proceedings of the
17th international conference on Parallel architectures and compilation
techniques, pages 260269. ACM, 2008.
[7] M. Horowitz, E. Alon, D. Patil, S. Naffziger, R. Kumar, and
K. Bernstein. Scaling, power, and the future of cmos. In Electron
Devices Meeting, 2005. IEDM Technical Digest. IEEE International,
pages 7pp. IEEE, 2005.
[8] I. Kuon and J. Rose. Measuring the gap between fpgas and
asics. Computer-Aided Design of Integrated Circuits and Systems, IEEE
Transactions on, 26(2):203215, 2007.
[9] C. Lattner and V. Adve. Llvm: A compilation framework for
lifelong program analysis & transformation. In Code Generation and
Optimization, 2004. CGO 2004. International Symposium on, pages
7586. IEEE, 2004.
[10] A. Putnam, A. M. Caulfield, E. S. Chung, D. Chiou, K. Constan-
tinides, J. Demme, H. Esmaeilzadeh, J. Fowers, G. P. Gopal, J. Gray,
et al. A reconfigurable fabric for accelerating large-scale datacenter
services. In Computer Architecture (ISCA), 2014 ACM/IEEE 41st
International Symposium on, pages 1324. IEEE, 2014.
[11] O. Segal, M. Margala, S. R. Chalamalasetti, and M. Wright. High
level programming for heterogeneous architectures. arXiv preprint
arXiv:1408.4964, 2014.
[12] D. Soderman and Y. Panchul. Implementing c designs in hard-
ware: a full-featured ansi c to rtl verilog compiler in action. In
Verilog HDL Conference and VHDL International Users Forum, 1998.
IVC/VIUF. Proceedings., 1998 International, pages 2229. IEEE, 1998.
[13] J. A. Stuart and J. D. Owens. Multi-gpu mapreduce on gpu
clusters. In Parallel & Distributed Processing Symposium (IPDPS),
2011 IEEE International, pages 10681079. IEEE, 2011.
[14] B. Sukhwani, H. Min, M. Thoennes, P. Dube, B. Brezzo, S. Asaad,
and D. E. Dillenberger. Database analytics: A reconfigurable-
computing approach. IEEE Micro, 34(1):1929, 2014.
[15] K. H. Tsoi and W. Luk. Axel: a heterogeneous cluster with
fpgas and gpus. In Proceedings of the 18th annual ACM/SIGDA
international symposium on Field programmable gate arrays, pages
115124. ACM, 2010.
[16] J. H. Yeung, C. Tsang, K. H. Tsoi, B. S. Kwan, C. C. Cheung, A. P.
Chan, and P. H. W. Leong. Map-reduce as a programming model
for custom computing machines. In Field-Programmable Custom
Computing Machines, 2008. FCCM08. 16th International Symposium
on, pages 149159. IEEE, 2008.

1556-6056 (c) 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.